eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My goal is to concurrently crawl URLs from a queue. Based on the crawling result, the queue may be extended. Here is the MWE: import queue from concurrent.futures import ThreadPoolExecutor import time def get(url): # let's assume that the HTTP magic happens here time.sleep(1) return f'data from {url}' def crawl(url, url_queue: queue.Queue, result_queue: queue.Queue): data = get(url) result_queue.put(data) if 'more' in url: url_queue.put('url_extended') url_queue = queue.Queue() result_queue = queue.Queue() for url in ('some_url', 'another_url', 'url_with_more', 'another_url_with_more', 'last_url'): url_queue.put(url) with ThreadPoolExecutor(max_workers=8) as executor: while not url_queue.empty(): url = url_queue.get() executor.submit(crawl, url, url_queue, result_queue) while not result_queue.empty(): data = result_queue.get() print(data)  In this MWE, two URLs require another crawl: 'url_with_more' and 'another_url_with_more'. They are added to the url_queue while crawling. However, this solution ends before those two 'more' URLs are processed; after running, the url_queue remains to have two entries. How can I make sure that the ThreadPoolExecutor does not exit too early? Have I misunderstood ThreadPoolExecutor?

Answer: You have a race condition where the check for more work to submit happens before new tasks are added from the other thread, you need to not exit the threadpool until you wait on all submitted jobs, and then check if there is more work to submit. task_queue = queue.Queue() with ThreadPoolExecutor(max_workers=8) as executor: # exit when no task pending or to be scheduled while not (url_queue.empty() and task_queue.empty()): # submit new tasks while not url_queue.empty(): url = url_queue.get() task_queue.put(executor.submit(crawl, url, url_queue, result_queue)) # wait for one task if not task_queue.empty(): task_queue.get().result() # process result here if needed.  You might want to use concurrent.futures.wait with FIRST_COMPLETED instead of waiting on the first task in the queue if tasks have varying times. tasks_to_wait = set() with ThreadPoolExecutor(max_workers=8) as executor: # exit when no task pending or to be scheduled while not (url_queue.empty() and len(tasks_to_wait) == 0): # submit new tasks while not url_queue.empty(): url = url_queue.get() tasks_to_wait.add(executor.submit(crawl, url, url_queue, result_queue)) # wait for one task if len(tasks_to_wait) != 0: done, tasks_to_wait = concurrent.futures.wait(tasks_to_wait, None, concurrent.futures.FIRST_COMPLETED)
Question: I want to import a module from a subpackage so I went here Relative importing modules from parent folder subfolder and since it was not working I read all the literature here on stack and found a smaller problem that I can reproduce but cannot solve. I want to use relative imports because I don't wanna deal with sys.path etc and I don't wanna install every module of my project to be imported everywhere. I wanna make it work with relative imports. My project structure: project/ __init__.py bar.py foo.py main.py  bar.py: from .foo import Foo class Bar(): @staticmethod def get_foo(): return Foo()  foo.py: class Foo(): pass  main.py: from bar import Bar def main(): f = Bar.get_foo() if __name__ == '__main__': main()  I am running the project code from terminal with python main.py and I get the following: Traceback (most recent call last): File "**omitted** project/main.py", line 1, in <module> from bar import Bar File "**omitted** project/bar.py", line 1, in <module> from .foo import Foo ImportError: attempted relative import with no known parent package  Why am I getting this error? It seems that bar doesn't recognize project as the parent package but:  __init__.py is in place bar.py is being run as a module not a script since it is called from main and not from the command line (so __package__ and __name__ should be in place to help solve the relative import Relative imports for the billionth time)  Why am I getting this error? What am I getting wrong? I have worked for a while just adding the parent of cwd to the PYTHONPATH but I wanna fix this once and for all.

Answer: You should be running your project from the parent of project dir as: $ python -m project.main # note no .py  This tells python that there is a package named project and inside it a module named main - then relative and absolute imports work correctly - once you change the import in main in either of from .bar import Bar # relative from project.bar import Bar # absolute
Question: Example dataframe, also available as a fiddle: import pandas as pd d = { "year": [2021, 2021, 2021, 2021, 2022, 2022, 2022, 2023, 2023, 2023, 2023], "type": ["A", "B", "B", "A", "A", "B", "A", "B", pd.NA, "B", "A"], "observation": [22, 11, 67, 44, 2, 16, 78, 9, 10, 11, 45] } df = pd.DataFrame(d) df_pivot = pd.pivot_table( df, values="observation", index="year", columns="type", aggfunc="count" )  The pivot table produces the desired output by count (this is intentional, I do not want the sum of observations, I want a row count): >>> print(df_pivot) type A B year 2021 2 2 2022 2 1 2023 1 2  However, I would like to show the percentage divided into total for each row by types "A" and "B" (the values of the "type" column in the dataframe). Note that not all rows have a type, some are NA (one is NA in this sample data to illustrate this). It's fine to ignore these unpopulated values in calculations. This also means that the "total" may be different in each row and is based on the sum of counted values in each type (i.e., count of A + count of B for each year). I have tried multiple ways but it only seems to work when I isolate each specific type one at a time. I have not been able to figure out how to do it where it has similar output only showing the percentage of the total instead of the count. My lambda functions for aggfunc seem to result in incorrect values not reflective of the correct percentages. Example desired output: >>> print(df_desired_output) type A B year 2021 0.50 0.50 2022 0.66 0.33 2023 0.33 0.66  How do I get this desired output?

Answer: Division should do it: df_pivot.div(df_pivot.sum(1),axis=0).round(2) type A B year 2021 0.50 0.50 2022 0.67 0.33 2023 0.33 0.67
Question: I have a DataFrame with a code in a column. I want to extract the first digit from said code and add to a different column so I can use it to merge it with a different DF. My code is: df_a = df_a.assign(index_a = lambda x: int(str(x.code)[0]))  When I use: df_a = df_a.assign(index_a = lambda x: x.code)  This works and I get a new DF with the extra "Code" column and the entire code. If do any operations here like x.code + 1 or x.code * 5 it works. Then, when I try to convert each code to a string by doing: df_a = df_a.assign(index_a = lambda x: str(x.code))  Instead of getting each row with a string code, all rows receive the value of the entire column converted into a massive string. I had a similar problem in the past trying to navigate lambda functions and learned that as long as I did x + 0 before converting, everything worked okay, but this time it's not working. I'm obviously doing something wrong, but I can't figure it out.

Answer: it's so because str(x.code) convert the whole column each time, try this instead df_a['code'] = df_a['code'].apply(str)
Question: I have similar problem of merging-multiple-files-based-on-the-common-column as https://superuser.com/questions/1245094/merging-multiple-files-based-on-the-common-column. I am very near to the solution but I am new to python. I need help with the tweaking of code for joining multiple files. My IDs and columns for individual file look like: File1.txt id SRR1071717 chr1:15039:-::chr1:15795:- 2 chr1:15948:-::chr1:16606:- 6  File2.txt id SRR1079830 chr1:11672:+::chr1:12009:+ 10 chr1:11845:+::chr1:12009:+ 7 chrY:9756574:+::chrY:9757796:+ 0  My desired output id SRR1071717 SRR1079830 chr1:15039:-::chr1:15795:- 2 0 chr1:15948:-::chr1:16606:- 6 0 chr1:11672:+::chr1:12009:+ 0 10 chr1:11845:+::chr1:12009:+ 0 7 chrY:9756574:+::chrY:9757796:+ 0 0  My code: Matrix.py import sys columns = [] data = {} ids = set() for filename in sys.argv[1:]: with open(filename, 'rU') as f: key = next(f).strip().split()[1] columns.append(key) data[key] = {} for line in f: if line.strip(): id, value = line.strip().split() try: data[key][int(id)] = value except ValueError as exc: raise ValueError( "Problem in line: '{}' '{}' '{}'".format( id, value, line.rstrip())) ids.add(int(id)) print('\t'.join(['ID'] + columns)) for id in sorted(ids): line = [] for column in columns: line.append(data[column].get(id, '0')) print('\t'.join([str(id)] + line))  I ran a python code as shown but it's not working correctly (being new to python). Current Output (two lines only!). python3 matrix.py File\*.txt  Current output id SRR1071717 SRR1079830 chrY:9756574:+::chrY:9757796:+ 0 0 

Answer: Using any awk: $ cat tst.awk FNR == 1 { ++numCols } { if ( !($1 in ids2rows) ) { rows2ids[++numRows] = $1 ids2rows[$1] = numRows } rowNr = ids2rows[$1] vals[rowNr,numCols] = $2 } END { for ( rowNr=1; rowNr<=numRows; rowNr++ ) { id = rows2ids[rowNr] printf "%s", id for ( colNr=1; colNr<=numCols; colNr++ ) { val = ( (rowNr,colNr) in vals ? vals[rowNr,colNr] : 0 ) printf "%s%s", OFS, val } print "" } }   $ awk -f tst.awk File1.txt File2.txt id SRR1071717 SRR1079830 chr1:15039:-::chr1:15795:- 2 0 chr1:15948:-::chr1:16606:- 6 0 chr1:11672:+::chr1:12009:+ 0 10 chr1:11845:+::chr1:12009:+ 0 7 chrY:9756574:+::chrY:9757796:+ 0 0
Question: I have multiple dataframes that are structured as follows:     Molecule Name Molecular weight Score Population Score Population Error     A 100 12 15 0.2   B 205 0.4 17 0.8   C 367 17 11 0.82   D 510 9 19.6 0.1        Molecule Name Molecular weight Score Population Score Population Error     A 100 20 15 0.2   B 205 16 17 0.8   E 367 11 11 0.82   F 780 11 12 0.5     Imagine I had multiple dataframes where the molecule names, weights, and score can vary. But the population score and population error are the same across all dataframes. I want to merge all the dataframes. The unique identifer is the Molecular weight. So if there are multiple dataframes which contain a molecular weight of 100, the scores of each get saved - take note of molecule C from the first dataframe and molecule E from the second - they have the same molecular weight so should be combined. I essentially want something that comes out like this:     Molecule Name Molecular weight Score1 Score2 Population Score Population Error     A 100 12 20 15 0.2   B 205 0.4 16 17 0.8   C 367 17 11 11 0.82   D 510 9 0 19.6 0.1   F 780 0 11 12 0.5     I have tried just simple merges but it always creates a new row for Molecule name E. I want to merge on Molecular weight. If the Molecular weight isn't in the original data frame then add a new row and populate the 0s. I hope this makes sense. Any and all help will be appreciated.

Answer: You could use a custom concat with de-duplication of the Score columns and groupby.first to get the first molecule name: dfs = [df1, df2] # could be more than 2 input DataFrames out = (pd.concat([d.set_index(['Molecular weight', 'Population Score', 'Population Error']) .rename(columns={'Score': f'Score{i}'}) for i, d in enumerate(dfs, start=1)], axis=1) .groupby(level=0, axis=1).first() .fillna(0, downcast='infer') .reset_index() )  Variant for future versions of pandas: dfs = [df1, df2] out = (pd.concat([d.set_index(['Molecular weight', 'Population Score', 'Population Error']) .rename(columns={'Score': f'Score{i}'}) for i, d in enumerate(dfs, start=1)], axis=1) .T.groupby(level=0).first().T .fillna(0) .reset_index() )  Output:  Molecular weight Population Score Population Error Molecule Name Score1 Score2 0 100 15.0 0.20 A 12.0 20 1 205 17.0 0.80 B 0.4 16 2 367 11.0 0.82 C 17.0 11 3 510 19.6 0.10 D 9.0 0 4 780 12.0 0.50 F 0.0 11
Question: I have a panda database with several columns, I want to make two types of filters based on a list: 1 - Filter rows that contain at least one item in the list, regardless of the column 2 - Filter rows that contain all items in the row. For example, column 1 might contain one element, and column 2 might contain another element An example of the database I have df = pd.DataFrame({'A' : ["Bonjour (coouse); je teste", "La nuit (horrible)", "La troisième voie"], 'B' : ["La première voie", "Bonjour (coouse); ou pas", "La nuit (horrible)"]}) df I've tried this filter, it only works per column, whereas I want to apply on all columns. It doesn't matter what the column is, if a row contains that value, then I display it. df.loc[df['A'].apply(lambda x: any([k in x for k in ["Bonjour (coouse)", "La troisième voie"]]))] The second solution I'd like to have is the & condition: all the values in the list must be contained in one of the columns. For example, the first value might be in column 1 and the second value in the list might be in column 2

Answer: Since you want to match substrings, I would use a regex approach. Filter rows that contain at least one item in the list, regardless of the column Here we craft a regex, extract the substrings and keep the rows with at least a non-NA value import re target = ["Bonjour (coouse)", "La troisième voie"] pattern = '(%s)' % '|'.join(map(re.escape, target)) # '(Bonjour\\ \\(coouse\\)|La\\ troisième\\ voie)' out = df[df.apply(lambda s: s.str.extract(pattern, expand=False)) .notna().any(axis=1)]  regex demo Filter rows that contain all items in the row We can craft the same regex, but instead count the number of different values that we have extracted, to have all it should match the length of the list: import re target = ["Bonjour (coouse)", "La troisième voie"] pattern = '(%s)' % '|'.join(map(re.escape, target)) # '(Bonjour\\ \\(coouse\\)|La\\ troisième\\ voie)' out = df[df.apply(lambda s: s.str.extract(pattern, expand=False)) .nunique(axis=1).eq(len(target))]  Alternatively, for the last step, using set operations (less efficient): out = df[df.apply(lambda s: s.str.extract(pattern, expand=False)) .agg(set, axis=1).eq(set(target))]  intermediate df.apply(lambda s: s.str.extract(pattern, expand=False)) A B 0 Bonjour (coouse) NaN 1 NaN Bonjour (coouse) 2 La troisième voie NaN
Question: Why can I no longer sort my imports in VS Code. I have changed zero settings and sometimes it just does not appear. 

Answer: They have recently changed this command to "Organize Imports" Comment that says it is ready for October update  https://github.com/microsoft/vscode-python/issues/22147#issuecomment-1751173694  Comment that says the new command  https://github.com/microsoft/vscode-python/issues/22147#issuecomment-1767140242
Question: What I want to achieve is something similar to this:  This is the code I added: from pandas import * from matplotlib.pyplot import * from math import * import numpy as num import pandas as pd import matplotlib import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D %matplotlib inline import seaborn as sns import statistics users = pd.read_table("/content/drive/MyDrive/Colab Notebooks/Datasets/users.dat",sep="::",names=['UserID','Gender','Age','Occupation','ZipCode'],encoding='latin-1') users['Age'].value_counts() sns.countplot(users["Age"])  And this is what I ended up with instead:  I tried using a different file to make sure that the data was correct, I also tried making sure that I didn't misspell anything incorrectly and tried single quotation marks just to get the same result. IDK what's the cause of the problem, unless its the data itself. BTW here's the users file if you're interested: https://drive.google.com/file/d/10iF0gKWAX8GfzOTgCrtgUHMjtcqcvv3R/view

Answer: In this code, the issue is due to countplot() which may be related to the distribution of the data. Countplot() visualizes data by counting occurrences in each age group, so if some age groups have very few instances, they might not appear on the plot. To address this, you can arrange the age groups in ascending order and adjust the figure size to ensure the visibility of all age groups on the plot. Try this code: users = pd.read_table("/content/drive/MyDrive/Colab Notebooks/Datasets/users.dat", sep="::", names=['UserID', 'Gender', 'Age', 'Occupation', 'ZipCode'], encoding='latin-1') age_counts = users['Age'].value_counts().sort_index() plt.figure(figsize=(10, 6)) sns.countplot(users["Age"], order=age_counts.index) plt.show()
Question: My VS Code was working fine, I have a pyenv environment with a specific python version and installed dependencies which I was using and all was good and suddenly it stopped recognizing all imports. They are all whited out. Also I noticed that the Sort Imports option disappeared from the context menu options I have when I right click. I have not changed anything in VS Code, any idea what might be wrong? Current VS Code Python extension version 2023.18.0

Answer: These seem to be fresh issues coming from VSCode and Python extension for VSCode. After doing a lot of digging around I think both of these issues come from very recent changes from VSCode. I mean they were just released this month (October 2023). So regarding the Sort Imports option VSCode just stopped supporting it this month, see reference and ticket opened on VSCode github here To be able to still get automatic sorting of our imports we need to explicitly install the isort extension in VSCode and Use the Organize Imports command with sortkey Shift + Alt + O. If someone wants to automate it completely you can add automatic import sorting on saving. Go to Preferences (Ctrl + Shift + P) search for Open User Settings (JSON). In the json file add this or modify the "[python]" settings section if already exists:  "[python]": { ...Other settings... "editor.formatOnSave": true, "editor.codeActionsOnSave": { "source.organizeImports": true }, },  Now if you restart VSCode you should get automated sorting of imports on save. Also see some visual instructions here. Regarding the issue of imports not being recognized I downgraded the VSCode Python extension to a previous version 2023.2.0 and it worked.
Question: I can't seem to figure out what the issue is here, I am running version 0.28.1:  from what I have read I should be using ChatCompletion rather than Completion as that's what gpt-4 and 3.5-turbo supports. response = openai.ChatCompletion.create( prompt=question, temperature=0, max_tokens=3700, top_p=1, frequency_penalty=0, presence_penalty=0, stop=None, model="gpt-4", )  Looking at other answers I can also tell you that my file isn't named openai.py or anything like that. Thanks for any help in advance.

Answer: First of all, be sure you have an up-to-date OpenAI package version. If not, upgrade the OpenAI package. Python: pip install --upgrade openai  NodeJS: npm update openai   The code posted in your question above has a mistake. The Chat Completions API doesn't have the prompt parameter as the Completions API does. Instead, it has the messages parameter. See the official OpenAI documentation. Try the following: import os import openai openai.api_key = os.getenv("OPENAI_API_KEY") completion = openai.ChatCompletion.create( model = "gpt-4", messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello!"} ] ) print(completion.choices[0].message)
Question: I'm trying to install fasttext using pip install fasttext in python 3.11.4 but I'm running into trouble when building wheels. The error reads as follows:  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.37.32822\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for fasttext Running setup.py clean for fasttext Failed to build fasttext ERROR: Could not build wheels for fasttext, which is required to install pyproject.toml-based projects  I've searched the web and most hits indicated that the error has something to do with the build tools of visual studio (which the error above also indicated). I've installed/updated all my build tools and I've also installed the latest SDK as suggested here, but the error persists. Has anyone solved this problem before and can share any potential solution?

Answer: Using pip install fasttext-wheel instead solved the problem for me.
Question: I want to remove the non highlighted region (turn it white) from this image and just retain the highlighted portions. This is my image :  Getting Highlighted Region  import cv2 import numpy as np image = cv2.imread('0002.jpg') gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) lower_green = (30, 90, 90) upper_green = (100, 255, 255) green_mask = cv2.inRange(cv2.cvtColor(image, cv2.COLOR_BGR2HSV), lower_green, upper_green) _, binary = cv2.threshold(gray, 128, 255, cv2.THRESH_BINARY_INV) binary = cv2.bitwise_not(binary) preserved_highlights = cv2.bitwise_and(image, image, mask=binary) image[green_mask == 0] = [255, 255, 255] #cv2.imwrite('image.jpg', image) result = cv2.add(preserved_highlights, image) cv2.imwrite('final.jpg', result)  OUTPUT1  Increasing Clarity As you can see the output is quite blurry, after applying some other operations :  image = cv2.imread('final.jpg') gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) alpha = 1.5 beta = 0 enhanced = cv2.convertScaleAbs(gray, alpha=alpha, beta=beta) sharpening_filter = np.array([[-1, -1, -1], [-1, 9, -1], [-1, -1, -1]]) sharpened = cv2.filter2D(enhanced, -1, sharpening_filter) # Save the final result cv2.imwrite('output_enhanced02.jpg', sharpened)  The best I could get was FINAL OUTPUT  Question Is there a way I can retain just the highlighted part like it is originally and just remove the non highlighted part? OR if that's not possible How can make my output look more readable and clear ? Thank you

Answer: I changed your code a little bit according to Christoph Rackwitz's comment. Here is the result import cv2 as cv import numpy as np image = cv.imread('Q0aCT.jpg', cv.IMREAD_COLOR) lower_green = (0, 100, 0) upper_green = (200, 255, 200) green_mask = cv.inRange(image, lower_green, upper_green) kernel = np.ones((5, 5), np.uint8) openning = cv.morphologyEx(green_mask, cv.MORPH_OPEN, kernel, iterations=1) kernel = np.ones((9, 9), np.uint8) closing = cv.dilate(openning, kernel, iterations=3) preserved_highlights = cv.bitwise_and(image, image, mask=closing) gray = cv.cvtColor(preserved_highlights, cv.COLOR_RGB2GRAY) _, preserved_text = cv.threshold(gray, 100, 255, cv.THRESH_BINARY_INV) preserved_text = cv.bitwise_and(preserved_text, preserved_text, mask=closing) cv.imwrite('final.png', 255 - preserved_text)  Steps to get the result  Using cv.inRange to find highlighted text Use opening morphology to remove salt noise Use close morphology to fill the black text inside the highlighted area Use a binary threshold to extract those black areas  Final Result