eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am attempting to run a multiprocessed job within a larger Python class. In a simple form, the class looks as following: class Thing: def test(self): with mp.Pool() as p: yield from p.map(str, range(20))  When I import this class to a script such as: from x import Thing t = Thing() for item in t.test(): print(item)  I run into the commonly known issue: RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable.  My question is, how do I guard against this behaviour, without requiring the user of my class to write if __name__ == "__main__":  every time they run my class function? Is there a way of performing this inside the class definition? I have tried writing from x import Thing t = Thing() if __name__ == "__main__": for item in t.test(): print(item)  which solves the issue, but I do not want this to be the way a user interfaces this class.

Answer: Since the value of __name__ in the main module of a spawned child process is '__mp_main__' as discussed here, you can place a guard in the function that spawns child processes by checking if any of the ancestors' frames has a __name__ in the global namespace with a value of '__mp_main__', in which case the current process is a child and the execution should stop so that no more child would be spawned: # x.py import sys import multiprocessing as mp class Thing: def test(self): frame = sys._getframe(1) while frame: if frame.f_globals['__name__'] == '__mp_main__': return frame = frame.f_back with mp.Pool() as p: yield from p.map(str, range(20))  EDIT: An even easier way to guard against a child process is to validate that the current process name is 'MainProcess': import multiprocessing as mp class Thing: def test(self): if mp.current_process().name == 'MainProcess': with mp.Pool() as p: yield from p.map(str, range(20))
Question: I am trying to change the color of the ticklines in my plot, where I would like to assign the colors based on a list of strings with color codes. I am following the following approach, but I cannot see why that does not work: import numpy as np import matplotlib.pyplot as plt x = [0, 1, 2, 3, 4, 5] y = np.sin(x) y2 = np.tan(x) fig = plt.figure() ax1 = fig.add_subplot(2, 1, 1) ax1.plot(x, y) ax2 = fig.add_subplot(2, 1, 2) ax2.plot(x, y2) colors = ['b', 'g', 'r', 'c', 'm', 'y'] ax1.set_xticks(x) for tick, tickcolor in zip(ax1.get_xticklines(), colors): tick._color = tickcolor plt.show()  Does anyone know the correct implementation of this?

Answer: As noted in comments, tick._color/tick.set_color(tickcolor) isn't working due to a bug: Using tick.set_markeredgecolor is the workaround, but it doesn't seem to be the only issue. ax1.get_xticklines() yields the actual ticks lines on every two items, you should thus only zip those: for tick, tickcolor in zip(ax1.get_xticklines()[::2], colors): tick.set_markeredgecolor(tickcolor)  Output:  NB. also changing the ticks width for better visualization of the colors. Full code: import numpy as np import matplotlib.pyplot as plt x = [0, 1, 2, 3, 4, 5] y = np.sin(x) y2 = np.tan(x) fig = plt.figure() ax1 = fig.add_subplot(2, 1, 1) ax1.plot(x, y) ax2 = fig.add_subplot(2, 1, 2) ax2.plot(x, y2) colors = ['b', 'g', 'r', 'c', 'm', 'y'] ax1.set_xticks(x) for tick, tickcolor in zip(ax1.get_xticklines()[::2], colors): tick.set_markeredgecolor(tickcolor) tick.set_markeredgewidth(4) plt.show()
Question: For a FastAPI Pydantic interface I want to be as tolerable as possible, such as receiving an integer for a string parameter and parse that integer to string: from pydantic import BaseModel class FooBar(BaseModel): whatever: str FooBar(whatever=12)  Gives: ValidationError: 1 validation error for FooBar whatever Input should be a valid string [type=string_type, input_value=12, input_type=int] For further information visit https://errors.pydantic.dev/2.4/v/string_type  I think in earlier versions this was possible.  Python 3.10 pydantic==2.4.2 pydantic_core==2.10.1 

Answer: This functionality was just reintroduced in version 2.4.0. You need to add the coerce_numbers_to_str configuration: from pydantic import BaseModel, ConfigDict class FooBar(BaseModel): model_config = ConfigDict(coerce_numbers_to_str=True) whatever: str
Question: In my webpage I have some pseudo elements. For example:  How can I find the ::after element using Playwright?

Answer: You can't get the pseudo-element because the browser doesn't expose it. But you can get the CSS properties using JavaScript. const someCSSPropertyValue = await page.locator('<selector>').first() .evaluate(el => window.getComputedStyle(el, ':after').someCSSProperty);  I bet that will be helpful if you need to test that some :after was applied.
Question: I have a TTS model and I want to combine audio. I need a way to convert the model output(numpy array) for pydub.AudioSegment to be able to combine audio This is the model output -  audio[0].data.cpu().numpy() = array([ 1.90522405e-04, 3.96589050e-04, 4.41852462e-04, ..., 1.13033675e-05, -1.63643017e-05, -2.01268449e-05], dtype=float32)  This is my function to combine the audio from pydub import AudioSegment from os.path import exists def creating_one_audio_file(audio): if exists("/content/audio_file.wav"): sound2 = AudioSegment.from_wav("/content/audio_file.wav") combined_sounds = audio + sound2 combined_sounds.export("/content/audio_file.wav", format="wav") else: combined_sounds = audio combined_sounds.export("/content/audio_file.wav", format="wav") creating_one_audio_file(audio[0].data.cpu().numpy()) 

Answer: You can rely on audiosegment (a wrapper of a pydub.AudioSegment) and its audiosegment.from_numpy_array method or borrow its underlying method implementation from https://github.com/MaxStrange/AudioSegment/blob/master/docs/api/audiosegment.py#L1145
Question: I have numpy array which represents my x-axis, ex. x = [1, 2, 3, 4, 5]  I want to get values in the middle of two adjacent element, so my example array should turn into x_interp = [1.5, 2.5, 3.5, 4.5]  Is there a fast and convinient way to do this using python/numpy

Answer: If you have a numpy array, just slice with a shift: x = np.array([1, 2, 3, 4, 5]) x_interp = (x[1:]+x[:-1])/2  Output: array([1.5, 2.5, 3.5, 4.5]) An alternative (and also more generic approach to get the mean of every N items) would be to use sliding_window_view: from numpy.lib.stride_tricks import sliding_window_view as swv N = 2 x_interp = swv(x, N).mean(axis=1)
Question: I try to force all users to have a date that starts at the earliest on the 1st of january 2023 with 3 use cases:  if there is already data on the 01/01/2023, I want to keep it and delete previous rows If there is no data on the 01/01/2023 but older rows, I want to update the date to 01/01/2023 if the 1st row of a user if after the 01/01/2023, keep it.  I guess I should start with a groupby() but then i'm not sure what to do? Source data::     USER Date Value     USER1 01/06/2022 1000   USER1 01/01/2023 1000   USER1 01/02/2023 1200   USER1 02/02/2023 1300   USER2 01/06/2021 1000   USER2 01/02/2023 1200   USER2 02/02/2023 1250   USER3 02/06/2023 1250     df = pd.DataFrame({"USER" : ['USER1', 'USER1', 'USER1', 'USER1', 'USER2', 'USER2', 'USER2', 'USER3'], "Date" :['01/06/2022', '01/01/2023', '01/02/2023', '02/02/2023', '01/06/2021', '01/02/2023', '02/02/2023', '02/06/2023'], "Value" :['1000', '1000', '1200', '1300', '1000', '1200', '1250', '1250'],} )  Target data::     USER Date Value      USER1 01/01/2023 1000 row before is deleted because there is a already a row on january 1st 2023   USER1 01/02/2023 1200    USER1 02/02/2023 1300    USER2 01/01/2023 1000 previous value "01/06/2021" is forced to 1st of january 2023   USER2 01/02/2023 1200    USER2 02/02/2023 1250    USER3 02/06/2023 1250 keep the value as is, it's already after the 1st of january 2023    

Answer: If I understand correctly, you could force a minimal date with clip, then drop_duplicates keeping the latest row: # ensure datetime df['Date'] = pd.to_datetime(df['Date']) out = (df.assign(Date=df['Date'].clip(lower=pd.Timestamp('2023-01-01'))) .drop_duplicates(subset=['USER', 'Date'], keep='last') )  NB. I am assuming that the dates are already sorted (per USER). Output:  USER Date Value 1 USER1 2023-01-01 1000 2 USER1 2023-01-02 1200 3 USER1 2023-02-02 1300 4 USER2 2023-01-01 1000 5 USER2 2023-01-02 1200 6 USER2 2023-02-02 1250 7 USER3 2023-02-06 1250  Intermediates:  USER Date Value clip duplicated 0 USER1 2022-01-06 1000 2023-01-01 True 1 USER1 2023-01-01 1000 2023-01-01 False 2 USER1 2023-01-02 1200 2023-01-02 False 3 USER1 2023-02-02 1300 2023-02-02 False 4 USER2 2021-01-06 1000 2023-01-01 False 5 USER2 2023-01-02 1200 2023-01-02 False 6 USER2 2023-02-02 1250 2023-02-02 False 7 USER3 2023-02-06 1250 2023-02-06 False
Question: I have installed the a Python Package with pip install pyjwt[crypto] and pip freeze shows me different packages that was installed by that action. What do I need to put in my requirements.txt to correctly freeze the versions I am developing with? I dont want to put to much in the requirements.txt, only what is needed for rebuilding the environment later. Output pip: ... pyjwt[crypto] in ..\site-packages (2.8.0) cryptography>=3.4.0 in ..\site-packages (from pyjwt[crypto]) (41.0.4) cffi>=1.12 in ..\site-packages (from cryptography>=3.4.0->pyjwt[crypto]) (1.16.0) pycparser in ..\site-packages (from cffi>=1.12->cryptography>=3.4.0->pyjwt[crypto]) (2.21) ...  Output pip freeze:  ... cffi==1.16.0 cryptography==41.0.4 pycparser==2.21 PyJWT==2.8.0 ...  Is it ok to put only PyJWT[crypto]==2.8.0 into requirements.txt? When I put to much into requirements.txt I would make it harder to install additional packages in the future ... I think.

Answer: You can keep your requirements.txt flexible enough, and use a constraint files. This constraint file is basically the output of pip freeze, and pins all versions, including their dependencies. If you need to add a dependency or upgrade, just do this in requirements.txt and regenerate the constraint file. That way, you are guaranteed to have consistent deployment. Technically: # Create the constraint file pip freeze > constraint.txt # use it: pip install -c constraint.txt  Official doc: https://pip.pypa.io/en/stable/user_guide/#constraints-files
Question: I have a large DataFrame, on which I need to calculate the rolling row-wise weighted average. I know I can do the following: import numpy as np import pandas as pd df = pd.DataFrame(np.random.rand(20000, 50)) weights = [1/9, 2/9, 1/3, 2/9, 1/9] rolling_mean = df.rolling(5, axis=1).apply(lambda seq: np.average(seq, weights=weights))  The issue is that this takes around 40 seconds on my PC. Is there any way to speed up this computation?

Answer: Code Creating a new dataframe of multiplying df by weights[0], then shifting df by one and multiplying by weights[1], then shifting df by two and multiplying by weights[2], and repeating this process, then adding all of the created dataframes together, will speed up the process. sum([df.shift(num, axis=1) * w for num, w in enumerate(weights)])  it take 0.05986 sec
Question: I have a dataframe: dateRep day month year cases deaths country_name Land.area..sq..km. 0 2021-09-21 21 9 2021 1162 7 Austria 82520.0 1 2021-09-20 20 9 2021 1708 7 Austria 82520.0 2 2021-09-19 19 9 2021 2072 5 Austria 82520.0 3 2021-09-18 18 9 2021 2235 9 Austria 82520.0 4 2021-09-17 17 9 2021 2283 8 Austria 82520.0 ... ... ... ... ... ... ... ... ... 6145 2021-03-05 5 3 2021 4069 15 Sweden 407310.0 6146 2021-03-04 4 3 2021 4882 19 Sweden 407310.0 6147 2021-03-03 3 3 2021 4873 18 Sweden 407310.0 6148 2021-03-02 2 3 2021 6191 23 Sweden 407310.0 6149 2021-03-01 1 3 2021 668975 13086 Sweden 407310.0 6150 rows × 8 columns  And my target is to make a new dataframe which has only rows that follow specific condition in 'dateRep' column The condition is being a holiday (Sunday or Saturday) using .weekday() function. when I tried doing: df.loc[(df['dateRep'].weekday() == 5) or (df['dateRep'].weekday == 6)]  I got this error:  AttributeError: 'Series' object has no attribute 'weekday'  How do I sort out these rows?

Answer: Convert the dateRep to datetime Series and use .dt. accessor: df["dateRep"] = pd.to_datetime(df["dateRep"]) print(df.loc[(df["dateRep"].dt.weekday == 5) | (df["dateRep"].dt.weekday == 6)])  Prints:  dateRep day month year cases deaths country_name Land.area..sq..km. 2 2021-09-19 19 9 2021 2072 5 Austria 82520.0 3 2021-09-18 18 9 2021 2235 9 Austria 82520.0   OR: Use .isin() for shorter code: df.loc[df["dateRep"].dt.weekday.isin([5, 6])]
Question: I am dealing with a very large DataFrame with many index columns and I wish to convert a few columns from the index to regular columns. Below is a simplified example: df = pd.DataFrame( { 'col_a': [1,2,3], 'col_b': [4,5,6], 'index_1': ['a','b','c'], 'index_2': ['f','g','h'], 'index_to_column': [True,False,False], }, ).set_index(['index_1', 'index_2', 'index_to_column'])  In the example above I would like index_to_column to be dropped from the index and become a normal column Is it possible to perform this operation without calling df.reset_index().set_index(['index_1', 'index_2']) as when dealing with very large numbers of columns and multiple columns to drop I find it gets confusing very quickly.

Answer: Just specify a level(s) to be reset: df.reset_index(2)    index_to_column col_a col_b index_1 index_2 a f True 1 4 b g False 2 5 c h False 3 6   To reset multiple selective indices pass a list of levels positions to df.reset_index like: df.reset_index(level=[0, 2])  or a list of level names: df.reset_index(level=['index_to_column', 'index_1'])
Question: So I am trying to build a weekly Data import to Unity Catalog in Databricks. I use python. There is no problem overwriting the table in case it exists: %sql Use catalog some_catalog dfTarget #some pandas dataframe df_sparkTarget=spark.createDataFrame(dfTarget) df_sparkTarget.write.format("delta").mode("overwrite").saveAsTable(database+"."+table)  But before I ovverwrite anything I would like to check for the existence of this table: if spark.catalog.tableExists( database+"."+table): print("Table exists") else: print("Table does not exist")  This returns the following error py4j.security.Py4JSecurityException: Method public boolean org.apache.spark.sql.internal.CatalogImpl.tableExists(java.lang.String) is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl  Where can I whitelist this? Maybe just for this one cluster and not for all.

Answer: Yes I have found much the same for now until there is documented ways to do this I am using this. Catalog functionality only seems to work for the hivemetastore def schema_exists(catalog:str, schema_name:str): query = spark.sql(f""" SELECT 1 FROM {catalog}.information_schema.schemata WHERE schema_name = '{schema_name}' LIMIT 1""") return query.count() > 0 def table_exists(catalog:str, schema:str, table_name:str): query = spark.sql(f""" SELECT 1 FROM {catalog}.information_schema.tables WHERE table_name = '{table_name}' AND table_schema='{schema}' LIMIT 1""", ) return query.count() > 0 schema_exists("catalog_name","schema_name") table_exists("catalog_name","schema_name", "table_name")
Question: I need to receive messages from a specific topic, and send them to my channel. there are no problems with sending, but when I try to receive a message from a group, I cannot understand which topic the message is from. (it is not necessary to use telethon) import asyncio import telethon from telethon import TelegramClient, events api_id = api_hash = "" client = TelegramClient("session_name", api_id, api_hash) client.start() @client.on(events.NewMessage([-1001941512580])) async def main(event): print(event) client.run_until_disconnected() 

Answer: In v1, the topic is inside the MessageReplyHeader: msg = event.message if msg.reply_to and msg.reply_to.forum_topic: topic_id = msg.reply_to.reply_to_top_id
Question: I see references to both psycopg2 and psycopg3, but no clear guidance wrt a roadmap for transitioning between the two. I see that over time there is a large body of SO questions regarding psycopg2. Is psycopg3 intended to be a replacement for psycopg2? Has there been a significant uptake of this version? Will there be a long-lived version of psycopg2? Are there any compelling reasons to choose one version over the other?

Answer: From the documentation of psycopg3:  Psycopg 3 is a newly designed PostgreSQL database adapter for the Python programming language. Psycopg 3 presents a familiar interface for everyone who has used Psycopg 2 or any other DB-API 2.0 database adapter, but allows to use more modern PostgreSQL and Python features, such as:  Asynchronous support COPY support from Python objects A redesigned connection pool Support for static typing Server-side parameters binding Prepared statements Statements pipeline Binary communication Direct access to the libpq functionalities   From a glance, psycopg3 appears to support more modern python and postgresql features like typing and async. Doing so likely required a lot of backwards-incompatible changes from psycopg2, hence the new version and forked development.
Question: Although I thought this question should be duplicated, I couldn't find the proper answer. I have some problems merging multiple dataframes sequentially. For example, I have four dataframes as below: df1 = pd.DataFrame({'source': ['A', 'A', 'A', 'B', 'B', 'C', 'C'], 'target': ['1', '2', '3', '4', '5', '6', '7']}) df2 = pd.DataFrame({'source': ['A', 'A'], 'temp': ['a', 'b']}) df3 = pd.DataFrame({'source': ['B', 'B'], 'temp': ['c', 'd']}) df4 = pd.DataFrame({'source': ['C'], 'temp': ['e']})  And I'd like to merge the dataframe as below: # source target temp #0 A 1 a #1 A 1 b #2 A 2 a #3 A 2 b #4 A 3 a #5 A 3 b #6 B 4 c #7 B 4 d #8 B 5 c #9 B 5 d #10 C 6 e #11 C 7 e  To do so, I tried to run the code, but it returned unexpected results. #Trial 1 dfs = pd.merge(df1, df2, on='source', how='left') dfs = pd.merge(dfs, df3, on='source', how='left') # new column was created with prefix, but I want to keep the three columns; source, target, temp #Trial 2 dfs = pd.merge(df1, df2, on='source', how='left') dfs['temp']=dfs.set_index('source')['temp'].fillna(df3.set_index('source')['temp'].to_dict()).values # it only fills the fixed number of NaN value, but there are some exception; one NaN in dfs, multiple values in other df3 or df4 #Trial 3 dfs = pd.merge(df1, df2, on='source', how='left') dfs[dfs['source']=='B']['temp']=pd.merge(df1, df3, on='source', how='left')['temp'].dropna() # it didn't change the dfs 

Answer: This is not a simple merge. You want to concat the df2,df3,df4, then merge with df1: df1.merge(pd.concat([df2,df3,df4]).drop_duplicates(), on='source')  Output:  source target temp 0 A 1 a 1 A 1 b 2 A 2 a 3 A 2 b 4 A 3 a 5 A 3 b 6 B 4 c 7 B 4 d 8 B 5 c 9 B 5 d 10 C 6 e 11 C 7 e
Question: I have two lists: list_1 = ['08667\nST 403', '08667\nST 403', '08667\nST 403'] list_2 = ['12233\nFION', '12233\nFION', '12233\nFION', '12233\nFION', '31147\nARAB\nP1454']  I want to be able to extract the names after the '\n' keyword. I am using this code: def parse_string(string): string = string.rsplit('\n', 1)[1] return string list_1 = ['08667\nST 403', '08667\nST 403', '08667\nST 403'] list_2 = ['12233\nFION', '12233\nFION', '12233\nFION', '12233\nFION', '31147\nARAB\nP1454'] new_list_1 = [] new_list_2 = [] for i in range(len(list_1)): new_list_1.append(parse_string(list_1[i])) for i in range(len(list_2)): new_list_2.append(parse_string(list_2[i]))  For the list_1, it works fine: new_list_1 ['ST 403', 'ST 403', 'ST 403']  but for the list_2: new_list_2 ['FION', 'FION', 'FION', 'FION', 'P1454']  So, the last element is P1454 instead of ARAB. How can I automate this to work for all cases?

Answer: You can use list comprehension. It will work for both the lists. list_1 = ['08667\nST 403', '08667\nST 403', '08667\nST 403'] list_2 = ['12233\nFION', '12233\nFION', '12233\nFION', '12233\nFION', '31147\nARAB\nP1454'] [x.split('\n')[1] for x in list_1] #['ST 403', 'ST 403', 'ST 403'] [x.split('\n')[1] for x in list_2] #['FION', 'FION', 'FION', 'FION', 'ARAB']   Now if you want P1454 instead of ARAB you can do: [x.split('\n')[2] if len(x.split('\n'))>2 else x.split('\n')[1] for x in list_2] #['FION', 'FION', 'FION', 'FION', 'P1454']  This will work for list_1 as well: [x.split('\n')[2] if len(x.split('\n'))>2 else x.split('\n')[1] for x in list_1] #['ST 403', 'ST 403', 'ST 403']   Universal Solution for both lists both values: [x.split('\n')[1:] for x in list_2] #[['ST 403'], ['ST 403'], ['ST 403']] [x.split('\n')[1:] for x in list_2] #[['FION'], ['FION'], ['FION'], ['FION'], ['ARAB', 'P1454']]
Question: I have a simple Flask based project in Pycharm. I am trying to debug this by right clicking and selecting debug option. But keeps getting below error: Connected to pydev debugger (build 232.9559.58) * Serving Flask app 'app' * Debug mode: on WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8000 * Running on http://192.168.0.29:8000 Press CTRL+C to quit * Restarting with stat C:\Python311\python.exe: can't open file 'C:\\Program': [Errno 2] No such file or directory Process finished with exit code 2  I am unable to understand why and what it is looking for in 'C:\\Program'. I am able to run the file perfectly fine and only having issues in debug. What can I try next?

Answer: This appears to be a bug in Pycharm. Multiple issues have been raised for the same in the JetBrains bug tracker.  FLASK_DEBUG=1 breaks debugger when Python/PyCharm installation path has spaces Flask Debug session cannot be started in PyCharm Flask debugger is not working on Windows because of white space in installation directory if application is specified flask server run failed because of space character in file path Can not run flask project in debug mod  You can follow this comment(or this comment) to get a temporary workaround.
Question: I have a numpy 1D array: import numpy as np arr = np.array([1, 1, 3, -2, -1, 2, 0, 2, 1, 1, -3, -1, 2])  I want split it into another two-dimensional array, based on changes in positive and negative values of array's elements(0 is placed in the range of positive values). But the original order of elements should be maintained. The desired result is: new_arr = [[1, 1, 3], [-2, -1], [2, 0, 2, 1, 1], [-3, -1], [2]] 

Answer: You could use array_split, diff, nonzero: np.array_split(arr, np.nonzero(np.diff(arr>=0))[0]+1)  Ouptut: [array([1, 1, 3]), array([-2, -1]), array([2, 0, 2, 1, 1]), array([-3, -1]), array([2])]  Intermediates: # arr>0 [ True True True False False True False True True True False False True] # np.diff(arr>=0) [False False True False True False False False False True False True] # np.nonzero(np.diff(arr>=0))[0]+1 [ 3 5 10 12]  And for lists as output: out = list(map(list, np.array_split(arr, np.nonzero(np.diff(arr>=0))[0]+1)))  Output: [[1, 1, 3], [-2, -1], [2, 0, 2, 1, 1], [-3, -1], [2]]  Or using itertools.groupby: from itertools import groupby out = [list(g) for _,g in groupby(arr, key=lambda x: x>=0)]  Output: [[1, 1], [3, -2], [-1, 2, 0, 2, 1], [1, -3], [-1, 2]]  comparison of approaches