eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a huge dataframe (2 million rows) in which a certain column has a string representations of a list of dictionaries (it is the school history of several people). So, what I'm trying to do is parsing this data to a new dataframe (because the relation is going to be 1 person to many schools). However, my first option was to loop over the dataframe with itertuples(). Too slow! The first few rows look like this: list_of_dicts = { 0: '[]', 1: "[{'name': 'USA Health', 'subject': 'Residency, Internal Medicine, 2006 - 2009'}, {'name': 'Ross University School of Medicine', 'subject': 'Class of 2005'}]", 2: "[{'name': 'Physicians Medical Center Carraway', 'subject': 'Residency, Surgery, 1957 - 1960'}, {'name': 'Physicians Medical Center Carraway', 'subject': 'Internship, Transitional Year, 1954 - 1955'}, {'name': 'University of Alabama School of Medicine', 'subject': 'Class of 1954'}]" } df_dict = pd.DataFrame.from_dict(list_of_dicts, orient='index', columns=['school_history'])  What I thought about, was to have a function and then apply it to the dataframe: def parse_item(row): eval_dict = eval(row)[0] school_df = pd.DataFrame.from_dict(eval_dict, orient='index').T return school_df df['column'].apply(lambda x: parse_item(x))  However, I'm not able to figure out how to generate a dataframe bigger than original (due to situations of multiple schools to one person). From those 3 rows, the idea is to have this dataframe (that has 5 rows from 2 rows): 

Answer: This does the trick using your sample data (thanks for the performance tip in comments): list_df = df_dict.school_history.map(ast.literal_eval) exploded = list_df[list_df.str.len() > 0].explode() final = pd.DataFrame(list(exploded), index=exploded.index)  This produces the following: In [54]: final Out[54]: name subject 1 USA Health Residency, Internal Medicine, 2006 - 2009 1 Ross University School of Medicine Class of 2005 2 Physicians Medical Center Carraway Residency, Surgery, 1957 - 1960 2 Physicians Medical Center Carraway Internship, Transitional Year, 1954 - 1955 2 University of Alabama School of Medicine Class of 1954  This will probably not be super fast given the amount of data, but parsing a dictionary of strings with nested objects inside will probably be pretty slow no matter what. You're probably better off parsing the file upstream first, then converting to pandas.
Question: For example, given an array arr = np.array([1,2,3,2,3,7,2,3,4])  there are 9 elements. I want to get sum every 3 elements: [6, 12, 9]  Is there any numpy api I can use?

Answer: If your arr can be divided into groups of 3, i.e. has length 3*k, then: arr.reshape(-1,3).sum(axis=-1) # array([ 6, 12, 9])  In the general case, bincounts: np.bincount(np.arange(len(arr))//3, arr) # array([ 6., 12., 9.])
Question: Assume I have a file content.txt, with the content Ex nihilo nihil fit. I am willing to replace it with Ex nihilo nihil est. The code is: with open("content.txt", "r+") as f: content = f.read() content = content.replace("fit", "est") print(content) f.write(content) f.close()  After that, the content of the file becomes: Ex nihilo nihil fit Ex nihilo nihil est  Why? The only thing I need is Ex nihilo nihil est. What is the correct code?

Answer: r+ opens the file and puts a pointer at the first byte. When you use f.read() the pointer moves to the end of the file, so when you try to write it starts at the end of the file. To move back to the start (so you can override), use f.seek(0): with open("text.txt", "r+") as f: content = f.read() content = content.replace("fit", "est") print(content) f.seek(0) f.truncate(0) f.write(content) f.close()
Question: I have two dataframes like this: DATE MAX_AMOUNT MIN_AMOUNT MAX_DAY MIN_DAY RATE 01/09/2022 20 15 10 5 0.01 01/09/2022 25 20 15 10 0.02 03/09/2022 30 10 5 3 0.03 03/09/2022 40 30 20 5 0.04 04/09/2022 10 5 10 1 0.05 ID DATE AMOUNT DAY 1 01/09/2022 18 7 2 01/09/2022 22 11 3 01/09/2022 30 20 4 03/09/2022 35 10 5 04/09/2022 35 10  I want to bring the RATE values to the second df in accordance with the DATE. Also, the AMOUNT and DAY values in the relevant DATE must be within the appropriate range (MAX_AMOUNT & MIN_AMOUNT, MAX_DAY & MIN_DAY). Desired output like this: ID DATE AMOUNT DAY RATE 1 01/09/2022 18 7 0.01 2 01/09/2022 22 11 0.02 3 01/09/2022 30 20 4 03/09/2022 35 10 0.04 5 04/09/2022 35 10  Could you please help me about this?

Answer: Use merge first with filter columns by Series.between and then use Series.map for RATE column with first matched ID - added DataFrame.drop_duplicates: df = df2.merge(df1, on='DATE') df = (df[df['AMOUNT'].between(df['MIN_AMOUNT'], df['MAX_AMOUNT']) & df['DAY'].between(df['MIN_DAY'], df['MAX_DAY'])]) df2['RATE'] = df2['ID'].map(df.drop_duplicates('ID').set_index('ID')['RATE']) print (df2) ID DATE AMOUNT DAY RATE 0 1 01/09/2022 18 7 0.01 1 2 01/09/2022 22 11 0.02 2 3 01/09/2022 30 20 NaN 3 4 03/09/2022 35 10 0.04 4 5 04/09/2022 35 10 NaN
Question: I have been trying to figure out a way to transform this dataframe. It contains only two columns; one is timeseries data and the other is event markers. Here is an example of the initial dataframe: df1 = pd.DataFrame({'Time': ['1:42 AM','2:30 AM','3:29 AM','4:19 AM','4:37 AM','4:59 AM','5:25 AM','5:33 AM','6:48 AM'], 'Event': ['End','Start','End','Start','End','Start','Start','End','Start']})  This is how I would like it to look after transformation: df2 = pd.DataFrame({'Start': ['', '2:30 AM', '4:19 AM', '4:59 AM', '5:25 AM', '6:48 AM'], 'End': ['1:42 AM', '3:29 AM', '4:37 AM', '', '5:33 AM', '']})  Essentially, I want to make the event markers the new columns, with the start and end times paired up down the table as they happen. This example includes both exceptions that will sometimes happen:  First row of data is an 'End' marker (data gets truncated by date). Sometimes there will not be an 'End' marker for a specific 'Start' marker (job failed or wasn't complete at time the report was run).  I looked at pivot and pivot_table but I wasn't able to add an index that got the output I wanted. Pretty sure this should be possible, I just am not an expert with dataframes yet.

Answer: Try: df1["tmp"] = df1["Event"].eq("Start").cumsum() df1 = df1.pivot(index="tmp", columns="Event", values="Time").fillna("") df1.columns.name, df1.index.name = None, None print(df1[["Start", "End"]])  Prints:  Start End 0 1:42 AM 1 2:30 AM 3:29 AM 2 4:19 AM 4:37 AM 3 4:59 AM 4 5:25 AM 5:33 AM 5 6:48 AM
Question: I've got a question that I could really use some guidance with. I am simply trying to serve HTML from a directory that is not the same directory as the server itself. Ideally, I'd like to move one level above the server's directory, and serve a file located there. When I try that, I get a 404. class Server(SimpleHTTPRequestHandler): def do_GET(self): ... some other stuff ... elif self.path == "/manage": self.send_response(200) self.path = "/manage.html" return SimpleHTTPRequestHandler.do_GET(self) else: f = self.send_head() if f: try: self.copyfile(f, self.wfile) finally: f.close()  ^this code serves the file, if it's in the same directory as the server. If I then move manage.html upward (where I'd like it to live), and try stuff like '../manage.html', or an absolute path, I get a 404. Am I bumping into like, some built-in directory traversal mitigation? If so, is there a way I can disable it? All of this is local, security isn't really a problem. I could try a subdirectory, but if I start down that road, I'd have to rename & rearrange the entire directory structure, because the naming won't make sense. Thanks in advance! (Python 3.10.2) 64-bit, Windows.

Answer: This is a security feature as you mentioned. You wouldn't want users to be able to see all files of the server, would you? Starting with Python 3.7, the constructor of SimpleHTTPRequestHandler has a directory parameter (see docs: https://docs.python.org/3/library/http.server.html#http.server.SimpleHTTPRequestHandler). With that you can tell SimpleHTTPRequestHandler from where to server the files. You can either change that wherever you instantiate your server, i.e. Server(directory=...) or you can alter the init method of you Server class class Server(SimpleHTTPRequestHandler): def __init__(self, *args, **kwargs): super().__init__(*args, directory=..., **kwargs)  EDIT: I dug a little deeper, and here is where the sanitization happens https://github.com/python/cpython/blob/9a95fa9267590c6cc66f215cd9808905fda1ee25/Lib/http/server.py#L839-L847 # ... path = posixpath.normpath(path) words = path.split('/') words = filter(None, words) path = self.directory for word in words: if os.path.dirname(word) or word in (os.curdir, os.pardir): # Ignore components that are not a simple file/directory name continue path = os.path.join(path, word) # ...
Question: I have my API, and some endpoints need to forward requests to Celery. Idea is to have specific API service that basically only instantiates Celery client and uses send_task() method, and seperate service(workers) that consume tasks. Code for task definitions should be located in that worker service. Basicaly seperating celery app (API) and celery worker to two seperate services. I dont want my API to know about any celery task definitions, endpoints only need to use celery_client.send_task('some_task', (some_arguments)). So on one service i have my API, an on other service/host I have celery code base where my celery worker will execute tasks. I came across this great article that describes what I want to do. https://medium.com/@tanchinhiong/separating-celery-application-and-worker-in-docker-containers-f70fedb1ba6d and this post Celery - How to send task from remote machine? I need help on how to create routes for tasks from the API? I was expecting for celery_client.send_task() to have queue= keyword, but it does not. I need to have 2 queues, and two workers that will consume content from these two queues. Commands for my workers: celery -A <path_to_my_celery_file>.celery_client worker --loglevel=info -Q queue_1 celery -A <path_to_my_celery_file>.celery_client worker --loglevel=info -Q queue_2  I have also visited celery "Routing Tasks" documentation, but it is still unclear to me how to establish this communication.

Answer: Your API side should hold the router. I guess it's not an issue because it is only a map of task -> queue (aka send task1 to queue1). In other words, your celery_client should have task_routes like: task_routes = { 'mytasks.some_task': 'queue_1', 'mytasks.some_other_task': 'queue_2', }
Question: I have the following code: import tensorflow as tf print("Hello")  And the output is: This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. Hello # This is printed about 5 seconds after the message  I had looked into the meaning of the message in this thread and in this one, but failed to make it disappear (and to run any program in less than 5 seconds). Any help would be greatly appreciated.

Answer: The message is just telling you that certain optimizations are on for you by default and if you want even more optimizations you can recompile TF to get even more performant optimizations. By default they compile using AVX2 which isn’t the fastest AVX, but it is the most compatible. If you don’t need to enable those (which you probably don’t) then you can just ignore the informational message knowing that you are getting some optimizations for your runs utilizing the oneDNN CPU optimizations.
Question: seeing this on Ubuntu 22.04 which has python 3.10 pip3 install psycopg2-binary==2.8.5 Defaulting to user installation because normal site-packages is not writeable Collecting psycopg2-binary==2.8.5 Using cached psycopg2-binary-2.8.5.tar.gz (381 kB) Preparing metadata (setup.py) ... error error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [23 lines of output] running egg_info creating /tmp/pip-pip-egg-info-qc0tg_5p/psycopg2_binary.egg-info writing /tmp/pip-pip-egg-info-qc0tg_5p/psycopg2_binary.egg-info/PKG-INFO writing dependency_links to /tmp/pip-pip-egg-info-qc0tg_5p/psycopg2_binary.egg-info/dependency_links.txt writing top-level names to /tmp/pip-pip-egg-info-qc0tg_5p/psycopg2_binary.egg-info/top_level.txt writing manifest file '/tmp/pip-pip-egg-info-qc0tg_5p/psycopg2_binary.egg-info/SOURCES.txt' Error: pg_config executable not found. pg_config is required to build psycopg2 from source. Please add the directory containing pg_config to the $PATH or specify the full executable path with the option: python setup.py build_ext --pg-config /path/to/pg_config build ... or with the pg_config option in 'setup.cfg'. If you prefer to avoid building psycopg2 from source, please install the PyPI 'psycopg2-binary' package instead. For further information please check the 'doc/src/install.rst' file (also at <https://www.psycopg.org/docs/install.html>). [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. 

Answer: This is a known issue. Use the binary version instead: pip install psycopg2-binary EDIT: You dont have 'pg_config' which is part of libpq-dev on Ubuntu. Install it with sudo apt-get install libpq-dev and try abgain.
Question: I accidentally used a mutable default argument without knowing it. Is there a linter or tool that can spot this and warn me?

Answer: flake8-bugbear, Pylint, PyCharm, and Pyright can detect this:  Bugbear has B006 (Do not use mutable data structures for argument defaults).  Do not use mutable data structures for argument defaults. They are created during function definition time. All calls to the function reuse this one instance of that data structure, persisting changes between them.   Pylint has W0102 (dangerous default value).  Used when a mutable value as list or dictionary is detected in a default value for an argument.   Pyright has reportCallInDefaultInitializer.  Generate or suppress diagnostics for function calls, list expressions, set expressions, or dictionary expressions within a default value initialization expression. Such calls can mask expensive operations that are performed at module initialization time.  This does what you want, but be aware that it also checks for function calls in default arguments.  PyCharm has Default argument's value is mutable.  This inspection detects when a mutable value as list or dictionary is detected in a default value for an argument. Default argument values are evaluated only once at function definition time, which means that modifying the default value of the argument will affect all subsequent calls of the function.  Unfortunately, I can't find online documentation for this. If you have PyCharm, you can access all inspections and navigate to this inspection to find the documentation.
Question: I have mounted a storage bucket on a VM using the command: gcsfuse my-bucket /path/to/mount  After this I'm able to read files from the bucket in Python using Pandas, but I'm not able to write files nor create new folders. I have tried with Python and from the terminal using sudo but get the same error. I have also tried Using the key_file from the bucket: sudo mount -t gcsfuse -o implicit_dirs,allow_other,uid=1000,gid=1000,key_file=Notebooks/xxxxxxxxxxxxxx10b3464a1aa9.json <BUCKET> <PATH>  It does not through errors when I run the code, but still I'm not able to write in the bucket. I have also tried: gcloud auth login  But still have the same issue.

Answer: I ran into the same thing a while ago, which was really confusing. You have to set the correct access scope for the virtual machine so that anyone using the VM is able to call the storage API. The documentation shows that the default access scope for storage on a VM is read-only:  When you create a new Compute Engine instance, it is automatically configured with the following access scopes:  Read-only access to Cloud Storage: https://www.googleapis.com/auth/devstorage.read_only   All you have to do is change this scope so that you are also able to write to storage buckets from the VM. You can find an overview of different scopes here. To apply the new scope to your VM, you have to first shut it down. Then from your local machine execute the following command: gcloud compute instances set-scopes INSTANCE_NAME \ --scopes=storage-rw \ --zone=ZONE  You can do the same thing from the portal if you go to the settings of your VM, scroll all the way down, and choose "Set Access for each API". You have the same options when you create the VM for the first time. Below is an example of how you would do this:
Question: I just came across some python code with the following statement: if a==b in [c,d,e]: ...  It turns out that: >>> 9==9 in [1,2,3] False >>> 9==9 in [1,2,3,9] True >>> (9==9) in [1,2,3,9] True >>> 9==(9 in [1,2,3,9]) False >>> True in [1,2,3,9] True >>> True in [] False >>> False in [] False >>> False in [1,2,3] False  Am I right in assuming that a==b in [c,d,e] is equivalent to (a==b) in [c,d,e] and therefore only really makes sense if [c,d,e] is a list of True/False values? And in the case of the code I saw b is always in the list [c,d,e]. Would it then be equivalent to simply using a==b?

Answer:  Am I right in assuming that a==b in [c,d,e] is equivalent to (a==b) in [c,d,e]  No. Since both == and in are comparison operators, the expression a == b in [c, d, e]  is equivalent to (a == b) and (b in [c, d, e])  since all comparison operators have the same precedence but can be chained.  and therefore only really makes sense if [c,d,e] is a list of True/False values?  It can also make sense to check if a boolean value is contained in a list of integers. Since True is considered equivalent to 1, and False is considered equivalent to 0 (see The standard type hierarchy), the result of this check can even be True.
Question: I would like to create multiple columns which show the row-wise cumulative mean for grouped columns. Here is some sample data: import pandas as pd data = [[1, 4, 6, 10, 15, 40, 90, 100], [2, 5, 3, 11, 25, 50, 90, 120], [3, 7, 9, 14, 35, 55, 100, 120]] df = pd.DataFrame(data, columns=['a1', 'a2', 'a3', 'a4', 'b1', 'b2', 'b3', 'b4']) a1 a2 a3 a4 b1 b2 b3 b4 0 1 4 6 10 15 40 90 100 1 2 5 3 11 25 50 90 120 2 3 7 9 14 35 55 100 120  What I want is to generate new columns like this:  New column a1_2 is calculated by the mean of columns a1 and a2 row-wise. New column a1_3 is calculated by the mean of columns a1, a2 and a3 row-wise. New column a1_4 is calculated by the mean of columns a1, a2, a3 and a4 row-wise.  The same should happen for the grouped columns with b. Of course you can do this manually, but this is not ideal when you have too many variables. Here is the expected output: df['a1_2'] = df[['a1', 'a2']].mean(axis=1) df['a1_3'] = df[['a1', 'a2', 'a3']].mean(axis=1) df['a1_4'] = df[['a1', 'a2', 'a3', 'a4']].mean(axis=1) df['b1_2'] = df[['b1', 'b2']].mean(axis=1) df['b1_3'] = df[['b1', 'b2', 'b3']].mean(axis=1) df['b1_4'] = df[['b1', 'b2', 'b3', 'b4']].mean(axis=1) a1 a2 a3 a4 b1 b2 b3 b4 a1_2 a1_3 a1_4 b1_2 b1_3 b1_4 0 1 4 6 10 15 40 90 100 2.5 3.666667 5.25 27.5 48.333333 61.25 1 2 5 3 11 25 50 90 120 3.5 3.333333 5.25 37.5 55.000000 71.25 2 3 7 9 14 35 55 100 120 5.0 6.333333 8.25 45.0 63.333333 77.50  So I was wondering if there is some automatic way of doing this?

Answer: expanding.mean for c in ('a', 'b'): m = df.filter(like=c).expanding(axis=1).mean().iloc[:, 1:] df[m.columns.str.replace(r'(\d+)$', r'1_\1', regex=True)] = m  Result  a1 a2 a3 a4 b1 b2 b3 b4 a1_2 a1_3 a1_4 b1_2 b1_3 b1_4 0 1 4 6 10 15 40 90 100 2.5 3.666667 5.25 27.5 48.333333 61.25 1 2 5 3 11 25 50 90 120 3.5 3.333333 5.25 37.5 55.000000 71.25 2 3 7 9 14 35 55 100 120 5.0 6.333333 8.25 45.0 63.333333 77.50  Another option: out = [value.expanding(axis=1).mean() .rename(columns = lambda col: f"{col[0]}1_{col[1]}") for _, value in df.groupby(df.columns.str[0], axis = 1)] pd.concat([df]+out, axis = 1) a1 a2 a3 a4 b1 b2 b3 b4 a1_1 a1_2 a1_3 a1_4 b1_1 b1_2 b1_3 b1_4 0 1 4 6 10 15 40 90 100 1.0 2.5 3.666667 5.25 15.0 27.5 48.333333 61.25 1 2 5 3 11 25 50 90 120 2.0 3.5 3.333333 5.25 25.0 37.5 55.000000 71.25 2 3 7 9 14 35 55 100 120 3.0 5.0 6.333333 8.25 35.0 45.0 63.333333 77.50
Question: I have the following straightforward code: import matplotlib.pyplot as plt x = [1,2,3,4] y = [34, 56, 78, 21] plt.plot(x, y) plt.show()  But after changing my MacBook Pro to the M1 chip, I'm getting the following error: Traceback (most recent call last): File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/main.py", line 291, in <module> plt.plot(x, y) File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 2728, in plot return gca().plot( File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 2225, in gca return gcf().gca() File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 830, in gcf return figure() File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/_api/deprecation.py", line 454, in wrapper return func(*args, **kwargs) File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 771, in figure manager = new_figure_manager( File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 346, in new_figure_manager _warn_if_gui_out_of_main_thread() File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 336, in _warn_if_gui_out_of_main_thread if (_get_required_interactive_framework(_get_backend_mod()) and File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 206, in _get_backend_mod switch_backend(dict.__getitem__(rcParams, "backend")) File "/Users/freddy/PycharmProjects/TPMetodosNoParametricos/venv/lib/python3.8/site-packages/matplotlib/pyplot.py", line 266, in switch_backend canvas_class = backend_mod.FigureCanvas AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'  Why does the code throw this error? My matplotlib version is 3.6.0

Answer: i had the same problem today on a different machine in the same matplotlib version. I downgrade to Version 3.5.0 and now it works.
Question: I am trying to add a column of empty lists to a polars dataframe in python. My code import polars as pl a = pl.DataFrame({'a': [1, 2, 3]}) a.with_columns([pl.lit([]).alias('b')])  throws Traceback (most recent call last): File "<input>", line 1, in <module> a.with_columns([pl.lit([]).alias('b')]) File "/usr/local/lib/python3.10/site-packages/polars/internals/lazy_functions.py", line 767, in lit return pli.wrap_expr(pylit(item, allow_object)) ValueError: could not convert value '[]' as a Literal  How can I create this column?

Answer: This works for me. I wrote pl.Series() with empty lists [] as values: import polars as pl from polars import col df = pl.DataFrame({'a': [1, 2, 3]}) # .lazy() df = df.with_columns([ col('a'), pl.Series('empty lists', [[]], dtype=pl.List), pl.lit(None).alias('null column'), ]) print(df) # print(df.collect()) (in case of LazyFrame) ┌─────┬─────────────┬─────────────┐ │ a ┆ empty lists ┆ null column │ │ --- ┆ --- ┆ --- │ │ i64 ┆ list[f64] ┆ bool │ ╞═════╪═════════════╪═════════════╡ │ 1 ┆ [] ┆ null │ ├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 2 ┆ [] ┆ null │ ├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 3 ┆ [] ┆ null │ └─────┴─────────────┴─────────────┘