eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to update pandas dataframe using list. Dataframe with columns A, B, C A B C ------ 1 a F 2 b F 3 c F 4 d F 5 e F  I have 2 lists, one contains list of elements whose value needs to update from column B and second contains actual value to replace in column C. Elements to update from column B names=['a', 'd', 'e'] Values to replace in column C values=['T', 'T', 'G'] Output after update A B C ------ 1 a T 2 b F 3 c F 4 d T 5 e G  How to update the dataframe?

Answer: You can use boolean indexing combined with map: names = ['a', 'd', 'e'] values = ['T', 'T', 'G'] m = df['B'].isin(names) df.loc[m, 'C'] = df.loc[m, 'B'].map(dict(zip(names, values)))  Less efficient alternatives: df['C'] = df['B'].map(dict(zip(names, values))).fillna(df['C']) df['C'] = df['C'].mask(df['B'].isin(names), df['B'].map(dict(zip(names, values))))  Output:  A B C 0 1 a T 1 2 b F 2 3 c F 3 4 d T 4 5 e G
Question: I'm using python cloud functions in my firebase project. After initializing cloud functions, adding firebase-admin to the requirements.txt file worked, and I could test with firebase emulators:start and also successfully deploy with firebase deploy --only functions. The issue is when I try to add other packages. I added tldextract to requirements.txt and put import tldextract in main.py which causes ModuleNotFoundError: No module named 'tldextract' 127.0.0.1 - - [14/Jun/2023 00:24:10] "GET /__/functions.yaml HTTP/1.1" 500 - ⬢ functions: Failed to load function definition from source: FirebaseError: Failed to parse build specification  when I run firebase emulators:start or firebase deploy --only functions. It also seems like the venv folder is not being updated. I tried activating the venv and pip install -r requirements.txt which made the local execution work with firebase emulators:start, BUT after redeploying the functions, they're stilling failing in the cloud. I tried this with different packages to make sure it's not just this one specific package. But adding other pip packages to requirements.txt and importing them in main.py failed for all packages that I tested. What am I doing wrong?

Answer: The following solved the issue for me: Delete the venv folder created by firebase init functions. Create a new one as follows: python3.11 -m venv venv source venv/bin/activate pip3 install --upgrade pip python3.11 -m pip install -r requirements.txt  Now deploy with firebase deploy --only functions
Question: I'm trying to use BeautifulSoup to find the birth years of different authors. I'm working in VS Code, if that's relevant. This is my first attempt at web scraping so please explain things as clearly as possible For authors with wikipedia pages, I can successully find birth years using the following code: source_code = requests.get("a_wikipedia_url") plain_text = source_code.text soup = BeautifulSoup(plain_text, features="html.parser") finder = soup.find("span", {"class": "bday"}) if finder is not None: birth_year = finder.string[0:4] return birth_year  However when I try the same thing with google search for authors with no (English) wikipedia page, I just get None. After reading this question https://stackoverflow.com/questions/62466340/cant-scrape-google-search-results-with-beautifulsoup I added a User Agent response header to requests.get (I'm using Chrome Version 114.0.5735.134 (Official Build) (64-bit) and Windows 11 Home), but all it did was print None instead of giving my AttributeError: 'NoneType' object has no attribute 'string', which is what I was getting before adding the header. This is my code: headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.134 Safari/537.36"} source_code = requests.get("https://www.google.com/search?q=Guillermo+Saccomanno", headers=headers) plain_text = source_code.text soup = BeautifulSoup(plain_text, features="html.parser") google_finder = soup.find("span", {"class": "LrzXr kno-fv wHYlTd z8gr9e"}) print(google_finder.string)  The result is just None - no error message, but no text. I also tried with the header Chrome version as Chrome/114.0.0.0, which is what I found online. Still gives None. I'm not sure where I'm going wrong as the syntax is identical and I copied the class name from the page source? For this particular author, I would expect google_finder.string to be "9 June 1948 (age 75 years)".

Answer: If you want to parse the born date I'd chose different strategy: Find a <span> tag with text "Born:" and then next sibling. Also add hl=en parameter to URL to get english results: import requests from bs4 import BeautifulSoup url = 'https://www.google.com/search?q=Guillermo+Saccomanno&hl=en' headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0'} soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser') born = soup.select_one('span:-soup-contains("Born:") + span') print(born.text)  Prints: June 9, 1948 (age 75 years), Buenos Aires, Argentina
Question: I was trying to make my first Tkinter project in Python but it just shows me this: >>> from tkinter import * Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.11/tkinter/__init__.py", line 38, in <module> import _tkinter # If this fails your Python may not be configured for Tk ^^^^^^^^^^^^^^^ ImportError: libtk8.6.so: cannot open shared object file: No such file or directory  I am using Arch. I have tried pip install tkinter but it didn't help. Pls help

Answer: Try # pacman -S tk I had the same problem and that fixed it.
Question: I'm trying to retrieve feature names from a Keras model in a generalized way. I want to load a pretrained model and obtain its feature names, like this: labels = model.get_feature_names()  I'm looking for something that works with any Keras model, ideally as a method that takes a black box Keras model and returns the feature names that the model operates on. For example, in SKlearn I can do this: from sklearn.base import BaseEstimator if isinstance(model, BaseEstimator): return lambda m: { 'labels': m.feature_names_in_, }  Is there a Keras/TensorFlow equivalent? Any insight is greatly appreciated.

Answer: No, unlike sklearn there isn't a Keras/Tensorflow equivalent for obtaining feature names. This is because tensorflow-keras models focus more on the shape and dimensions of the input tensors rather than the individual features. The model utilizes the input features as tensors, regardless of name. With some effort you might be able to track your individual features by values. However, if your end objective is building a classifier out of tabular data (and not some complex Computer-Vision or NLP tasks), you can consider using alternatives such as XGBoost (or perhaps other classifiers from sklearn itself). These classifiers are feature focused and you'll be able to work with the feature_names. They are also better suited to tabular data due to their versatility and ability to generalize well even with limited samples/features.
Question: I have a code which draws lines for the teams according to their tournament position in each game week. Pretty much I managed to make it work, except 2 things:  For some reason a 4th (violet) line is drawn (teams are only 3) which goes from the top to the bottom throughout each game week. As I found out this line is drawn for every iteration (for each team) when plotting the lines. But why? Lines are drawn from x = 0 starting point, thus not aligning with the points (which are drawn correctly). Lines should be drawn from x = 1 starting point as well (according to their Game_week value). Output:   What have I missed? Example of the code: import pandas as pd import matplotlib.pyplot as plt import numpy as np df = pd.DataFrame([['Team1', 1, 1], ['Team1', 2, 2], ['Team1', 1, 3], ['Team1', 5, 4], ['Team1', 1, 5], ['Team2', 2, 1], ['Team2', 3, 2], ['Team2', 4, 3], ['Team2', 4, 4], ['Team2', 3, 5], ['Team3', 3, 1], ['Team3', 4, 2], ['Team3', 3, 3], ['Team3', 2, 4], ['Team3', 2, 5] ], columns=['Team', 'Position', 'Game_week']) positions = df['Position'] weeks = df['Game_week'] teams = df['Team'].unique() print(teams) # Coordinates: y = positions x = weeks print(y) print(x) fig, ax = plt.subplots() # Labels: plt.xlabel('Game weeks') plt.ylabel('Positions') plt.xlim(-0.2, 5.2) plt.ylim(0.8, 5.2) # Inverting the y-axis: plt.gca().invert_yaxis() # x, y ticks: xi = list(np.unique(x)) yi = list(np.unique(y)) plt.xticks(xi) plt.yticks(yi) # Colors for teams: colors = {'Team1': 'tab:red', 'Team2': 'tab:blue', 'Team3': 'blue'} # Points: plt.scatter(x, y, s=45, c=df['Team'].map(colors), zorder=2) # Lines between points: for i, (team, l) in enumerate(df.groupby('Team', sort=False)): plt.step(list(zip(l['Game_week'], l['Position'])), '-', color=colors[team], linewidth=8, alpha=0.2, zorder=1) print('step:', i, '; team:', [team]) print(l) plt.show() plt.close()  Thank you!

Answer: The issues is with the plt.step(), where you are using zip. As per documentation here, you just need to give the x and y values. Updating that line as below... for i, (team, l) in enumerate(df.groupby('Team', sort=False)): plt.step(l['Game_week'], l['Position'], ## No zip '-', color=colors[team], linewidth=8, alpha=0.2, zorder=1)  .. will give you this plot. Does this meet your needs? You may want to change xlim to start at 0.8 perhaps.
Question: I am trying to run my pytests in parallel using the pytest plugins parallel-0.1.1 and xdist-3.2.1 along with the --tests-per-worker n flag. I have a set of tests that require a preprocessing step which must be run in a critical section. This section is protected by a multiprocessing lock to avoid simultaneous execution by multiple workers. However, despite using the lock, the workers enter the critical section simultaneously, leading to synchronization problems. Here is a simplified version of the problematic code: Test code: import pytest @pytest.mark.parametrize("preprocess", ["config1"], indirect=True) def test_example1(preprocess): # Use the preprocessed data in the test print(f"Test using preprocessed data: {preprocess}") # do something with preprocess @pytest.mark.parametrize("preprocess", ["config2"], indirect=True) def test_example2(preprocess): # Use the preprocessed data in the test print(f"Test using preprocessed data: {preprocess}") # do something with preprocess  conftest.py file: import pytest import multiprocessing lock = multiprocessing.Lock() @pytest.fixture def preprocess(request): with lock: # critical section  Why is my lock not preventing simultaneous entry into the critical section when running the tests in parallel? How can I resolve this synchronization problem? I appreciate any assistance with this matter!

Answer: A library such as https://pypi.org/project/fasteners/ has better locking mechanisms for the goal you're trying to accomplish. You want a lock based around a file so that different processes don't create different locks. import fasteners lock = fasteners.InterProcessLock('path/to/lock.file') with lock: ... # exclusive access  Then your code becomes thread-safe / process-safe.
Question: For the following example, where it involves a self conditional join and a subsequent groupby/aggregate operation. It turned out that in such case, DuckDB gives much better performance than Polars (~10x on a 32-core machine). My questions are:  What could be the potential reason(s) for the slowness (relative to DuckDB) of Polars? Am I missing some other faster ways of doing the same thing in Polars?  import time import duckdb import numpy as np import polars as pl ## example dataframe rng = np.random.default_rng(1) nrows = 5_000_000 df = pl.DataFrame( dict( id=rng.integers(1, 1_000, nrows), id2=rng.integers(1, 10, nrows), id3=rng.integers(1, 500, nrows), value=rng.normal(0, 1, nrows), ) ) ## polars start = time.perf_counter() res = ( df.lazy() .join(df.lazy(), on=["id", "id2"], how="left") .filter( (pl.col("id3") > pl.col("id3_right")) & (pl.col("id3") - pl.col("id3_right") < 30) ) .group_by(["id2", "id3", "id3_right"]) .agg(pl.corr("value", "value_right")) .collect(streaming=True) ) time.perf_counter() - start # 120.93155245436355 ## duckdb start = time.perf_counter() res2 = ( duckdb.sql( """ SELECT df.*, df2.id3 as id3_right, df2.value as value_right FROM df JOIN df as df2 ON (df.id = df2.id AND df.id2 = df2.id2 AND df.id3 > df2.id3 AND df.id3 - df2.id3 < 30) """ ) .aggregate( "id2, id3, id3_right, corr(value, value_right) as value", "id2, id3, id3_right", ) .pl() ) time.perf_counter() - start # 18.472263277042657 

Answer: EDIT: 2023-7-18 The latest polars release has brought the difference down from 15x to 2x. polars v0.18.2 1125 polars v0.18.3 140 duckdb 0.8.2-dev1 75  Original answer Streaming engine The streaming API isn't as optimized yet. Polars is a younger project than DuckDB and we haven't got as many paid developers on the project. So give us time. Next release 0.18.3 will land a PR that can make a streaming groupby over 3.5x faster https://github.com/pola-rs/polars/pull/9346. That just shows how much we still have on the table on the streaming engine. That same optimization we still have to do for streaming joins. In short. Our streaming engine is in alpha stage. It is work in progress. Different algorithm Other that that, the duckdb query might also be using non-equi joins under the hood which we don't have yet in polars, so this query might not be as optimal for polars.
Question: I want to reverse the order of characters in each string element of a NumPy array. For example, given the following input: array(['2', '3', '5', '7', '11', '13', '17', '19', '23', '29', '31', '37', '41', '43', '47', '53', '59', '61', '67', '71', '73', '79', '83', '89', '97'], dtype='<U2')  I want to obtain the following output (without using Python for loop): array(['2', '3', '5', '7', '11', '31', '71', '91', '32', '92', '13', '73', '14', '34', '74', '35', '95', '16', '76', '17', '37', '97', '38', '98', '79'], dtype='<U2')  I know that I can use arr[::-1] to reverse the order of elements in a NumPy array, but that isn't the topic of this question, and np.array([e[::-1] for e in arr]) is inefficient and against the point of NumPy. The array was created using a vectorized version of the base conversion function np.vectorize(to_base_str). How can I reverse the order of characters in each string element of a NumPy array using vectorization? I have searched online but have not found a solution. Note that arr[..., ::-1] does not work for string elements in a NumPy array. (Code is mine, but I did use the "AI suggested edits" feature)

Answer: np.array([e[::-1] for e in arr]) is the straight forward way of doing this, and is NOT bad numpy. or bypass numpy entirely with [e[::-1] for e in arr.tolist()]. You could also do something similar with np.vectorize or np.frompyfunc. These might scale a bit better. 'vectorize' in numpy means using compiled methods (and operators) to do the necessary iterations in compiled code. Those are nearly all numeric operations. For strings, numpy uses Python string methods. It does not have its own compiled string operations. Even the np.char functions use python string methods. So there's no numpy equivalent to astr[::-1]. Some comparative times In [16]: timeit np.array([s[::-1] for s in arr]) 36.1 µs ± 151 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) In [17]: timeit np.array([s[::-1] for s in arr.tolist()]) 21.1 µs ± 76.5 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) In [18]: timeit [s[::-1] for s in arr.tolist()] 8.29 µs ± 23.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) In [20]: timeit np.vectorize(lambda s: s[::-1])(arr) 65.9 µs ± 165 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) In [21]: timeit np.frompyfunc(lambda s: s[::-1],1,1)(arr) 20.3 µs ± 76.5 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
Question: I have a simple function that inserts a Python dictionary into DuckDB. How can I insert it into my table without creating a temporary file? def save_to_duckdb(data): # Connect to the Duckdb database conn = duckdb.connect('nodes_log_duck.db') # Get the table name from the "name" field in the dictionary table_name = data.get('name') # Create a temp file file_name = table_name + str(int(time.time())) with open( file_name,"w") as file: json.dump(data,file) # Create the table if it doesn't exist conn.execute(f" CREATE TABLE IF NOT EXISTS {table_name} as SELECT * FROM read_json_auto({file_name});") # Insert the dictionary data into the table conn.execute(f"INSERT INTO {table_name} FROM (SELECT * FROM read_json_auto({file_name}))") # Commit the changes to the database and close the connection conn.commit() conn.close() 

Answer: It seems there is no way to insert a Python dictionary into DuckDB 0.8.1. I use Polars DataFrame for this, and based on a GitHub discussion in the DuckDB repository, someone suggested using fsspec, which works fine. Although using read_json with fsspec creates better data types for DuckDB tables. **fsspec** def save_to_duckdb(data, db_name): with duckdb.connect(db_name) as conn: # Get the table name from the "name" field in the dictionary table_name = data.get('name') if not table_name: return # Create a memory filesystem and write the dictionary data to it with fsspec.filesystem('memory').open(f'{table_name}.json', 'w') as file: file.write(json.dumps(data)) # Register the memory filesystem and create the table conn.register_filesystem(fsspec.filesystem('memory')) conn.execute(f"CREATE TABLE IF NOT EXISTS {table_name} AS SELECT * FROM read_json_auto('memory://{table_name}.json')") # Insert the data into the table conn.execute(f"INSERT INTO {table_name} SELECT * FROM read_json_auto('memory://{table_name}.json')")  **Plors** def save_to_duckdb(data, db_name): # Get the table name from the "name" field in the dictionary table_name = data.get('name') if table_name is None: return # Create a polars DataFrame from the data dictionary df = pl.DataFrame(data) # Connect to the Duckdb database and insert the DataFrame into the database with duckdb.connect(db_name) as con: con.execute(f"CREATE TABLE IF NOT EXISTS {table_name} AS SELECT * FROM df") con.execute(f"INSERT INTO {table_name} SELECT * FROM df") con.commit()  "Notice that this code inserts data into the database twice for the first time."
Question: I calculated the wind direction using Metpy. How can I extract the values and use the values as input in another part of my program? In the sample code below, I would like to display the direction as a numpy variable, so I can use it in my program. import metpy.calc as mpcalc import numpy as np # Make some fake data for us to work with np.random.seed(19990503) # So we all have the same data u = np.random.randint(0, 15, 10) * units('m/s') v = np.random.randint(0, 15, 10) * units('m/s') direction = mpcalc.wind_direction(u, v) print(direction) 

Answer: To convert a metpy result from a wind calculation to a numpy array, simply use the numpy np.array function. import metpy.calc as mpcalc from metpy.units import units import numpy as np np.random.seed(19990503) u = np.random.randint(0, 15, 10)*units("m/s") v = np.random.randint(0, 15, 10)*units("m/s") direction = mpcalc.wind_direction(u, v) direction_numpy = np.array(direction)
Question: Given the following numpy array: arr = np.array([0, 1, 2, 3, 4, 5])  what iterable would return sub-arrays of length x from arr? (Given that len(arr) is a multiple of x) x = 2 sub_arrays = [sub_arr for sub_arr in iterable(arr, x)]   sub_arrays = [ np.ndarray( [0, 1] ), np.ndarray( [2, 3] ), np.ndarray( [4, 5] ) ]  I know that array slicing is possible with start, stop, and step arguments, but that returns individual elements: x = 2 sub_elements = [sub_elem for sub_elem in arr[::x]]   sub_elements = [0, 2, 4] 

Answer: To iterate over a numpy array and obtain sub-arrays of a specific length, you can use the numpy.reshape function. By reshaping the array with the desired shape, you can obtain sub-arrays of the specified length. Here's an example: import numpy as np arr = np.array([0, 1, 2, 3, 4, 5]) x = 2 sub_arrays = np.reshape(arr, (-1, x))  In this example, np.reshape is used to reshape the array arr into sub-arrays of length x. The parameter -1 in the reshape function allows numpy to automatically determine the appropriate size for that dimension. The resulting sub_arrays will be a 2D numpy array containing the sub-arrays: array([[0, 1], [2, 3], [4, 5]])  You can also convert each sub-array into a separate ndarray if desired: sub_arrays = [np.array(sub_arr) for sub_arr in sub_arrays]  This will give you a list of numpy arrays as you specified: [array([0, 1]), array([2, 3]), array([4, 5])]  Have a good coding!
Question: Until recently, it was possible to generate sample dataframes in Pandas using functionality of pd.util.testing module: In [22]: import pandas as pd In [23]: pd.util.testing.makeMixedDataFrame() Out[23]: A B C D 0 0.0 0.0 foo1 2009-01-01 1 1.0 1.0 foo2 2009-01-02 2 2.0 0.0 foo3 2009-01-05 3 3.0 1.0 foo4 2009-01-06 4 4.0 0.0 foo5 2009-01-07  (see https://stackoverflow.com/a/65592210/22084711 for more examples) However, pd.util.testing is being deprecated. As far as I can tell, this deprecation is in favor of pd.testing. It does not include any of the functionality used for generating sample dfs (makeMixedDataFrame, makeMissingDataframe, etc.). Is this functionality being transferred to some other module? I looked but couldn't find anywhere else. I'd like to have an alternative that comes with Pandas and does not require additional dependencies like Seaborn, or downloading the dataframe from somewhere else. (I was going to ask on pandas' Github, but they require that all questions are being asked on SO first.)

Answer: Actually, there is two different testing modules (if we can say so). An official one (which is documented in the API with only four available functions as of 2.0.0+) and a second one (for internal use). So, I guess you're looking for the latter (i.e pandas._testing) : import pandas as pd #pd.__version__ #2.0.2 df = pd._testing.makeMixedDataFrame()  Output : print(df) A B C D 0 0.0 0.0 foo1 2009-01-01 1 1.0 1.0 foo2 2009-01-02 2 2.0 0.0 foo3 2009-01-05 3 3.0 1.0 foo4 2009-01-06 4 4.0 0.0 foo5 2009-01-07
Question: I'd like to shift a pcolor plot along the x direction. But I'm not sure how to do it, as it's not as simple as using plot with a vector that specifies the x values With this code: import matplotlib.pyplot as plt import numpy as np np.random.seed(0) Z = np.random.rand(6, 5) tk = list(range(0,10+1)) fig, ax = plt.subplots(figsize=(7.2, 2.3)) ax.pcolor(Z) ax.set_xticks(tk) plt.show()  It produced this plot:  However I want the heatmap shifted to the right to start at x = 2, for example, like the following plot:  What am I missing??? As a side question, if I swap lines 11 and 12 to: ax.set_xticks(tk) ax.pcolor(Z)  I get this plot with the x axis contracted to the range [0,5]. I'm not sure why setting ticks before adding pcolor would do that? 

Answer: One 'simple' way of achieving that is by just adding 2 empty (NaN) values to the Z: Z = np.insert(Z, 0, np.nan, axis=1) Z = np.insert(Z, 0, np.nan, axis=1)  Gives: