eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am writing a Python Jupyter notebook that does some NLP processing on Italian texts. I have installed spaCy 3.5.3 via Poetry and then attempt to run the following code: import spacy load_model = spacy.load('it_core_news_sm')  The import line works as expected, but running spacy.load produces the following error:  OSError: [E050] Can't find model 'it_core_news_sm'. It doesn't seem to be a Python package or a valid path to a data directory. The model name is correct as shown on https://spacy.io/models/it  After a web search, I see that a solution is to issue the following command: python3 -m spacy download it_core_news_sm  After running this command the above code works as expected, however, is there a more 'kosher' way of doing this via Poetry?

Answer: You can add a URL dependency. First edit your pyproject.toml file to add the following (note: the name used here should match the name of the package (i.e. it_core_news_sm): [tool.poetry.dependencies] it_core_news_sm = {url = "https://github.com/explosion/spacy-models/releases/download/it_core_news_sm-3.5.0/it_core_news_sm-3.5.0.tar.gz"}  Then run the corresponding add call: poetry add https://github.com/explosion/spacy-models/releases/download/it_core_news_sm-3.5.0/it_core_news_sm-3.5.0.tar.gz  All of the spaCy models can be found on spaCy's model releases GitHub page.
Question: I have Predicted values and Actual values, and I can calculate Mean Absolute Percentage Error by doing: abs(Predicted-Actual)/ Predicted *100  How do I calculate MASE with respect to my Predicted and Actual values?

Answer: The Mean Absolute Scaled Error (MASE) is calculated by subtracting the forecasted or predicted value from the actual value divided by the average forecast error where the actual value from the prior step is used as the prediction or forecast. Here is the equation as a list-comprehension: mase = numpy.mean([abs(Actual[i] - Predicted[i]) / (abs(Actual[i] - Actual[i - 1]) / len(Actual) - 1) for i in range(1, len(Actual))])  Here is the equation as a function: def MASE(Actual, Predicted): values = [] for i in range(1, len(Actual)): values.append(abs(Actual[i] - Predicted[i]) / (abs(Actual[i] - Actual[i - 1]) / len(Actual) - 1)) return numpy.mean(values)
Question: For example, I'm wondering how is it possible that scikit-learn is the name of a PyPi package while the actual Python module is named sklearn. The reason I'm asking is that I have a local Python package packageA that I can't upload to PyPi since that name happens to already be taken. I therefore wonder if I can upload it as packageB (which actually is available on PyPi)? If so, how can I do that?

Answer: The names on PyPi or the names you are useing when doing pip install NAME are Distribution Packages. The names you use when doing import NAME are Import Packages. One Distribution Package can have multiple Import Packages in it. Example As an example see this demo project bit-demo:  The name of the repositories (you find in the URL) is "bit-demo". The name of the Distribution Package (you would use via pip) is bitdemo. This is defined in the packages metadata via pyproject.toml file. There are two Import Packages in it named bitcli and bitgui. This is also defined in the package metadata.  Scikit-learn The project setup of Scikit-learn is bit more complex but you also can see it there.  The Distributon Package name is defined in setup.py Not absolute sure about how the Import Package name is defined. But I would say setup.cfg tells to look up sub-folders name. And there is scilearn.
Question: I don't understand how work output in my case. Could you explain to me what I'm doing wrong? task:(27. Remove Element)  Given an integer array nums and an integer val, remove all occurrences of val in nums in-place. The order of the elements may be changed. Then return the number of elements in nums which are not equal to val. Consider the number of elements in nums which are not equal to val be k, to get accepted, you need to do the following things:  Change the array nums such that the first k elements of nums contain the elements which are not equal to val. The remaining elements of nums are not important as well as the size of nums.  Return k.    My solution: class Solution: def removeElement(self, nums: List[int], val: int) -> int: if len(nums) > 1: nums = [u for u in nums if u != val] print("nums = ", nums) if val not in nums: return len(nums)  Testcase: nums =[0,1,2,2,3,0,4,2] val = 2 Results: Stdout (print in loop): nums = [0, 1, 3, 0, 4] Output: [0,1,2,2,3] Expected: [0,1,4,0,3]

Answer: The problem is that you are not removing the elements in-place, so nums outside the function is not affected. One option is to iterate over the list and remove items with pop. To avoid index out of range exception iterate over the list from end to start for i in range(len(nums) - 1, 0, -1): if nums[i] == val: nums.pop(i)
Question: I want to use Python to cut the transparent area of the image file. Please look at the picture below. image 1 (original image)  image 2 (result image)  The original image has alternating opaque and transparent areas. I want to transform the image consisting of 1-spaces-2-spaces-3-spaces-4 like the original image into 1-2-3-4. I tried the following code but failed to get the desired output typfrom PIL import Image import numpy as np def trim_transparency(image): # Convert image to np.array im = np.array(image) # Import alpha channels only alpha = im[:, :, 3] # Found a range of nonzero indexes on each axis non_empty_columns = np.where(alpha.max(axis=0)>0)[0] non_empty_rows = np.where(alpha.max(axis=1)>0)[0] # Found a range of nonzero indexes on each axis cropBox = (min(non_empty_rows), max(non_empty_rows), min(non_empty_columns), max(non_empty_columns)) image_data = Image.fromarray(im).crop(cropBox) return image_data image = Image.open("original_image.png") new_image = trim_transparency(image) new_image.save("new_image.png")e here 

Answer: You were very close - I would use Numpy indexing to gather the rows you want: from PIL import Image import numpy as np # Open image and ensure not palettised, make into Numpy array and select alpha channel im = Image.open('e0rRU.png').convert('RGBA') na = np.array(im) alpha = na[:, :, 3] # Find opaque rows non_empty_rows = np.where(alpha.max(axis=1)>0)[0] # Copy them to new image opaque = na[non_empty_rows,...] # Save Image.fromarray(opaque).save('result.png')
Question: I am attempting to write a specific value to a column in a pandas df depending on if another column does or does not contain certain text. I have 3 outputs in the destination columns: Distributor, OEM and End User. import pandas as pd df = pd.read_excel("Customer Records.xlsx") #Checking for distributor pricing tag df.loc[df['CustomerRoles'].str.contains('Discount-Distributor-STD'), 'Customer Type'] = 'Distributor' #Checking for OEM pricing tag df.loc[df['CustomerRoles'].str.contains('Discount-OEM-STD'), 'Customer Type'] = 'OEM'  Those lines of code work as intended. The last thing I need is for every other string that contains neither of those phrases to print "End User" in the 'Customer Type' column.

Answer: You can use np.select: conds = [ df['CustomerRoles'].str.contains('Discount-Distributor-STD'), df['CustomerRoles'].str.contains('Discount-OEM-STD') ] choices = ['Distributor', 'OEM'] df['Customer Type'] = np.select(condlist=conds, choicelist=choices, default='End User')
Question: import asyncio import random task_group: asyncio.TaskGroup | None = None async def coro1(): while True: await asyncio.sleep(1) print("coro1") async def coro2(): while True: await asyncio.sleep(1) if random.random() < 0.1: print("dead") assert task_group is not None task_group.cancel() # This function does not exist. else: print("Survived another second") async def main(): global task_group async with asyncio.TaskGroup() as tg: task_group = tg tg.create_task(coro1()) tg.create_task(coro2()) task_group = None asyncio.run(main())  In this example, coro1 will print "coro1" every second, coro2 has a 10% chance to cancel the entire TaskGroup, i.e., cancel both coro1 and coro2 and exit the async with block every second. The problem is I don't know how to cancel the task group. There is no TaskGroup.cancel() function.

Answer: I end up use another task to wrap the function that contains the TaskGroup and cancel that task instead. It works as desired. # Only show functions that has been changed main_task: asyncio.Task[None] | None = None async def coro2(): while True: await asyncio.sleep(1) if random.random() < 0.1: print("dead") assert main_task is not None main_task.cancel() main_task = None else: print("Survived another second") async def main2(): global main_task main_task = asyncio.create_task(main()) asyncio.run(main2())
Question: I have an existing index that is created using GPTVectorStoreIndex. However, when I am trying to add a new document to the existing index using the insert method, I am getting the following error : AttributeError: 'list' object has no attribute 'get_text' my code for updating the index is as follows : max_input_size = 4096 num_outputs = 5000 max_chunk_overlap = 256 chunk_size_limit = 3900 prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit) llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="gpt-3.5-turbo", max_tokens=num_outputs)) service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper) directory_path = "./trial_docs" file_metadata = lambda x : {"filename": x} reader = SimpleDirectoryReader(directory_path, file_metadata=file_metadata) documents = reader.load_data() print(type(documents)) index.insert(document = documents, service_context = service_context) 

Answer: I got it right, the mistake I was doing it was passing documents as a whole, which is a List object. The right way to update is as follows max_input_size = 4096 num_outputs = 5000 max_chunk_overlap = 256 chunk_size_limit = 3900 prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit) llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="gpt-3.5-turbo", max_tokens=num_outputs)) service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper) directory_path = "./trial_docs" file_metadata = lambda x : {"filename": x} reader = SimpleDirectoryReader(directory_path, file_metadata=file_metadata) documents = reader.load_data() print(type(documents)) for d in documents: index.insert(document = d, service_context = service_context)
Question: I'm a developer without an interest in benchmarking and I'm trying to decide whether I should use Flask or FastAPI to build some Python/Vue projects. I'm seeing stuff online about how FastAPI was faster than Flask because Flask was single-threaded or something like that, whereas FastAPI was async, but apparently more-recently Flask added async routes, and so now I'm wondering if FastAPI is still(?) faster than Flask. Has anyone done benchmarking tests comparing FastAPI to Flask async routes? I can't find any when I search Google.

Answer: According to a benchmark study by Miguel Grinberg, FastAPI can be faster or slower than async Flask, depending on the web server and the Flask async type. Generally Flask on a Greenlet powered WSGI server (Meinheld / Gevent) can offer comparable throughput as an async-first ASGI framework like FastAPI. Note that Grinberg is comparing the overall performance of three parts: the Framework, the Web server and the Web application. Here are the relevant frameworks' throughput in the three different scenarios he tried (a higher number is better):     Framework Web Server Type Test 1 Throughput Test 2 Throughput Test 3 Throughput     Flask Meinheld Async / Greenlet 1.43 5.27 1.06   Flask Gevent Async / Greenlet 1.22 4.54 1.01   FastAPI Uvicorn Async / Coroutine 1.21 4.33 1.02   Aioflas Uvicorn Async / Coroutine 1.11 - -   Flask uWSGI Sync 1.09 1.01 1.26   Flask Gunicorn Sync 1.00 (baseline) 1.00 (baseline) 1.00 (baseline)     * Meinheld is a WSGI server written in C. Grinberg points out in his article that the exact relative results between frameworks will depend on the particular load that the server is put under, but his takeaway is that there isn't a large difference between frameworks:  remember that the difference in performance between different frameworks or web servers isn't going to be very significant, so choose the tools that make you more productive!  More about Async approaches of Flask Flask adopts different approaches to async view. In Grinberg's benchmark tests, three approaches have been implemented: Aioflask (standard python ASGI with uvicorn), Greenlet in Meinheld and Gevent. Since the Aioflask test was not fully finished, only the result of Test 1 is available, which agrees with the Flask documentation:  Flask’s async/coroutine support is less performant than async-first frameworks due to the way it is implemented.  Be aware that the Greenlet approach requires a special way of writing asynchronous operations, instead of standard python coroutines. Therefore, existing asyncio codebase needs to be patched with adapters like greenletio. Gevent relies heavily on proper monkey patching the python standard libraries, database engines and other performance critical libraries. Grinberg addressed multiple times this in his blog:  Gevent tests did reasonably well in my benchmark and terribly in the original one. This is because the author forgot to patch the psycopg2 package so that it becomes non-blocking under greenlets.
Question: I want to write a regex that matches a string that may contain both words and digits and not digits only. I used this regex [A-z+\d*], but it does not work. Some matched samples: expression123 123expression exp123ression  Not matched sample: 1235234567544  Can you help me with this one? Thank you in advance

Answer: Lookarounds to the rescue! ^(?!\d+$)\w+$  This uses a negative lookahead construct and anchors, see a demo on regex101.com  Note that you could have the same result with pure Python code alone: samples = ["expression123", "123expression", "exp123ression", "1235234567544"] filtered = [item for item in samples if not item.isdigit()] print(filtered) # ['expression123', '123expression', 'exp123ression']  See another demo on ideone.com. With both approaches you wouldn't account for input strings like -1 or 1.0 (they'd be allowed).  Tests As the discussion somewhat arose, here's a small test suite for different sample sizes and expressions: import string, random, re, timeit class RegexTester(): samples = [] expressions_to_test = {"Cary": "^(?=.*\D)\w+$", "Jan": "^(?!\d+$)\w+$"} def __init__(self, sample_size=100, word_size=10, times=100): self.sample_size = sample_size self.word_size = word_size self.times = times # generate samples self.samples = ["".join(random.choices(string.ascii_letters + string.digits, k=self.word_size)) for _ in range(self.sample_size)] # compile the expressions in question for key, expression in self.expressions_to_test.items(): self.expressions_to_test[key] = {"raw": expression, "compiled": re.compile(expression)} def describe_sample(self): only_digits = [item for item in self.samples if all(char.isdigit() for char in item)] return only_digits def test_expressions(self): def regex_test(samples, expr): return [expr.search(item) for item in samples] for key, values in self.expressions_to_test.items(): t = timeit.Timer(lambda: regex_test(self.samples, values["compiled"])) print("{key}, Times: {times}, Result: {result}".format(key=key, times=self.times, result=t.timeit(100))) rt = RegexTester(sample_size=10 ** 5, word_size=10, times=10 ** 4) #rt.describe_sample() rt.test_expressions()  Which for a sample size of 10^5, a word size of 10 gave the comparable results for the both expressions: Cary, Times: 10000, Result: 6.1406331 Jan, Times: 10000, Result: 5.948537699999999  When you set the sample size to 10^4 and the word size to 10^3, the result is the same: Cary, Times: 10000, Result: 10.1723557 Jan, Times: 10000, Result: 9.697761900000001  You'll get significant differences when the strings consist only of numbers (aka the samples are generated only with numbers): Cary, Times: 10000, Result: 25.4842013 Jan, Times: 10000, Result: 17.3708319  Note that this is randomly generated text and due to the method of generating it, the longer the strings are, the less likely they are to consist only of numbers. In the end it will depend on the actual text inputs.
Question: I have a python project with tkinter. On this project I draw small squares over time. I noticed tkinter is slowing down as the number of square increases. Here is a simple example that draws 200 red squares on each iteration: import tkinter as tk import random import time WIDTH = 900 CELL_SIZE = 2 GRID_WIDTH = int(WIDTH / CELL_SIZE) CELL_PER_ITERATION = 200 SLEEP_MS = 50 root = tk.Tk() canvas = tk.Canvas(root, width=WIDTH, height=WIDTH, bg="black") canvas.pack() current_iteration = 0 cell_count = 0 previous_iteration_end = time.time() text = tk.Label(root, text=f"iteration {current_iteration}") text.pack() def draw_cell(x_grid, y_grid): x = x_grid * CELL_SIZE y = y_grid * CELL_SIZE canvas.create_rectangle(x, y, x + CELL_SIZE, y + CELL_SIZE, fill="red") def iteration(): global current_iteration global previous_iteration_end global cell_count current_iteration_start = time.time() for _ in range(CELL_PER_ITERATION): draw_cell( x_grid=random.randint(0, GRID_WIDTH), y_grid=random.randint(0, GRID_WIDTH), ) cell_count += 1 current_iteration_end = time.time() # duration of this iteration current_iteration_duration = current_iteration_end - current_iteration_start # duration between start of this iteration and end of previous iteration between_iteration_duration = current_iteration_start - previous_iteration_end current_iteration += 1 text.config(text=f"iteration {current_iteration} | cell_count: {cell_count} | iter duration: {int(current_iteration_duration*1000)} ms | between iter duration: {int(between_iteration_duration*1000)} ms") previous_iteration_end = current_iteration_end def main_loop(): iteration() root.after(ms=SLEEP_MS, func=main_loop) root.after(func=main_loop, ms=SLEEP_MS) root.mainloop()  Which gives (time data is written at the bottom of picture):  And after a few seconds:  So the time to execute an iteration stays constant. But between two iterations, the duration keeps increasing over time. I don't understand why tkinter is slowing down. Is it redrawing the entire canvas (so all already drawn squares) at each iteration ? Is there a way to avoid this slow down ? Note: This is an example, the real project i am working on looks like this: Slime Mold Simulation

Answer: When adding shapes, you are not just colorize pixels. You create graphics with contexts and store them into memory. Instead of abusing the Canvas paint a picture and show this picture in the Canvas this will be a lot faster and will give you more options to colorize your image that you rather want to draw.
Question: I'm learning Langchain and vector databases. Following the original documentation I can read some docs, update the database and then make a query. https://python.langchain.com/en/harrison-docs-refactor-3-24/modules/indexes/vectorstores/examples/pinecone.html I want to access the same index and query it again, but without re-loading the embeddings and adding the vectors again to the ddbb. How can I generate the same docsearch object without creating new vectors? # Load source Word doc loader = UnstructuredWordDocumentLoader("C:/Users/ELECTROPC/utilities/openai/data_test.docx", mode="elements") data = loader.load() # Text splitting text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) texts = text_splitter.split_documents(data) # Upsert vectors to Pinecone Index pinecone.init( api_key=PINECONE_API_KEY, # find at app.pinecone.io environment=PINECONE_API_ENV ) index_name = "mlqai" embeddings = OpenAIEmbeddings(openai_api_key=os.environ['OPENAI_API_KEY']) docsearch = Pinecone.from_texts([t.page_content for t in texts], embeddings, index_name=index_name) # Query llm = OpenAI(temperature=0, openai_api_key=os.environ['OPENAI_API_KEY']) chain = load_qa_chain(llm, chain_type="stuff") query = "que sabes de los patinetes?" docs = docsearch.similarity_search(query) answer = chain.run(input_documents=docs, question=query) print(answer) 

Answer: You need to access the existing index. In order to do this, you must know the name of the index, and what embeddings were used to create it. index_name = "mlqai" embeddings = OpenAIEmbeddings(openai_api_key=os.environ['OPENAI_API_KEY']) docsearch = Pinecone.from_existing_index(index_name, embeddings)  Documentation.
Question: Snakemake rules in standardized workflows run Python scripts using the script directive, such as this template rule: rule XXXXX: input: ..., output: ...., params: ..., conda: "../envs/python.yaml" script: "../scripts/XXXX.py"  Then in the script, it is possible to use snakemake object. However, the script is then tightly coupled with that rule, which seems a big disadvantage. Why is this approach preferred to the approach using shell that calls the script, such as in this rule? rule XXXXX: input: ..., output: ...., params: absolute_script_path = ..., # get argument1 = ..., conda: "../envs/python.yaml" shell: "python {params.absolute_script_path} {input} {params.argument1} > {output}"  In this approach, python script is decoupled from the Snakemake rule. Also it looks more cohesive, as called arguments are clear from the rule, not hidden in the script. I am only starting with writing Snakemake workflows, so I am just a beginner. I do not understand why the first approach is preferred (or used in standardized Snakemake workflows) to the second approach? Am I missing something? Are there some problems with the second approach? Thank you very much for answers!

Answer: The script approach is a bit more flexible in terms of the objects that the script can access via the params and other directives. If you follow the shell approach you might find it cumbersome to (re) define the argparse or other approaches to properly take account of the arguments passed via shell. It's going to be mostly boilerplate, but can get somewhat tedious. The notebook directive might be useful in scenarios that require interactive reproduction/development. All in all, there are no hard rules, and for a given workflow one approach might be more suitable/convenient than other approaches.