eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a very large python dictionary. I want to perform an element-wise averaging for each element in each list. Let's say: dict = { "a": [2,5,3], "b": [1,0,2], "c": [5,2,5] }  The output should be: [2.6 2.3 3.3] where each element is the average of all the elements at that index in all the lists in dictionary. This naive approach does not seem efficient because the dictionary is very large and so are the lists in dictionary. avg = np.zeros(3) for item in dict.values(): for x in range(0,3): avg[x] += item[x] for x in range(0,3): avg[x] /= 3 print(avg)  How to do this efficiently?

Answer: This seems 2-3 times faster than mozway's fastest (for size 1000x1000): avg = [sum(column) / len(column) for column in zip(*d.values())]  Benchmarked on Attempt This Online!:  19.09 ± 0.03 ms Kelly 26.33 ± 0.19 ms Kelly_fmean 51.09 ± 0.25 ms mozway_numpy 249.17 ± 1.17 ms bart_original 266.40 ± 5.85 ms mozway_mean Python: 3.11.4 (main, Sep 9 2023, 15:09:21) [GCC 13.2.1 20230801]  In mozway's test, bart_original was slower than mozway_mean, so let me try on two more systems: On replit.com:  16.58 ± 0.20 ms Kelly 19.62 ± 0.17 ms Kelly_fmean 34.98 ± 0.10 ms mozway_numpy 184.64 ± 0.25 ms bart_original 244.38 ± 0.78 ms mozway_mean Python: 3.10.11 (main, Apr 4 2023, 22:10:32) [GCC 12.2.0]  On trinket.io:  24.17 ± 0.50 ms Kelly 31.63 ± 0.64 ms Kelly_fmean 75.26 ± 1.96 ms mozway_numpy 261.52 ± 7.42 ms bart_original 289.71 ± 10.68 ms mozway_mean Python: 3.10.9 (main, Jan 23 2023, 22:32:48) [GCC 10.2.1 20210110]  Benchmark script: def Kelly(d): return [sum(column) / len(column) for column in zip(*d.values())] def Kelly_fmean(d): return list(map(fmean, zip(*d.values()))) def bart_original(dict): avg = np.zeros(1000) for item in dict.values(): for x in range(0,1000): avg[x] += item[x] for x in range(0,1000): avg[x] /= 1000 return avg def mozway_numpy(d): return np.array(list(d.values())).mean(axis=0).tolist() def mozway_mean(d): return list(map(mean, zip(*d.values()))) funcs = Kelly, Kelly_fmean, bart_original, mozway_numpy, mozway_mean import random from timeit import timeit, default_timer as timer from statistics import mean, fmean, stdev import numpy as np import sys d = {i: random.choices(range(10), k=1000) for i in range(1000)} times = {f: [] for f in funcs} def stats(f): ts = [t * 1e3 for t in sorted(times[f])[:5]] return f'{mean(ts):6.2f} ± {stdev(ts):4.2f} ms ' for _ in range(50): for f in funcs: t0 = timer() f(d) times[f].append(timer() - t0) for f in sorted(funcs, key=stats): print(stats(f), f.__name__) print('\nPython:', sys.version)
Question: I am using "tab10" palette because of its distinct colors blue, green, orange and red. k_clust = KMeans(n_clusters=4, random_state= 35, n_init=1).fit(df_normal) palette = sns.color_palette("tab10") sns.pairplot(new_df, hue="clusters", palette=palette)  The number of clusters are only 4 and the palette "tab10" has more than 4 colors. Is there a way to address this UserWarning? The output is: C:\Users\....\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\seaborn\axisgrid.py:1507: UserWarning: The palette list has more values (10) than needed (4), which may not be intended. func(x=vector, **plot_kwargs) 

Answer: The docs for color_palette() say that you can pass n_colors=4 to the call. Try this: ... palette = sns.color_palette("tab10", n_colors=4) # equal to n_clusters ...
Question: Recently I have started to work with python RavenDB client, and found out that documentation for the python client is not full. Official page even does not contain Python as an option for the doceumentation. Anyhow, official client is there and I wanted to understand if it contains something like "session.Advanced.UseOptimisticConcurrency", I tried to look into coding and did not find anything like this. Does anybody know about it? Mabye it is turned on by default? The docuemtnation for the python does not contain any hint.

Answer: This option can be set in your DocumentConventions. You can set a property of your DocumentStore with your custom document conventions. Then upon internal/automatic RequestExecutor creation, the conventions will be passed forward to be processed.
Question: I am trying to deploy a simple Flask app to Elastic beanstalk. I am able to deploy the sample version. However, I am currently struggling to deploy my own. My Python app is already named application.py and changed flask name in the code to "application" in the code. Inside my .ebextension files are the following: postgres.config: packages: yum: mariadb-devel: [] postgresql-devel: []  (edit: removed mariadb-devel. now says postgres is not available to install. More info below) python.config: commands: 01_update_pip: command: "pip install --upgrade pip setuptools" option_settings: aws:elasticbeanstalk:container:python: WSGIPath: application.py  I am using the AWS website for deployment. I am semi-new and still figuring it out. My directory looks like this:  server (root)  .ebextensions - postgres.config - python.config application.py requirements.txt   venv  in the logs some noticeable things: [INFO] Error occurred during build: Yum does not have postgresql-devel available for installation [ERROR] An error occurred during execution of command [app-deploy] - [PreBuildEbExtension] [WARN]could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size  I've tried running another command that seems to at least deploy in Amazon Linux 2 but with a bad gateway error. Sadly, I do not remember it any more. Plus, I would prefer if possible to stick to the newest version. If it's easier attempting it in a different way then that's OK. On Amazon Linux 2 (assuming name is Application.py): commands: 01_update_pip: command: "pip install --upgrade pip setuptools" option_settings: aws:elasticbeanstalk:container:python: WSGIPath: Application:application 

Answer: The documents for Elastic Beanstalk is not updated for AL2023 and still uses AL2, so don't just follow elastic beanstalk docs if you want to use AL2023. I don't think they have postgresql-devel package for AL2023, based on this link for installed package list. Also, they changed package manager from yum to dnf, even though yum should still work, I would recommend changing yum to dnf in your file. If you can, it is best if you ssh into deployed elastic beanstalk instance and type "dnf search" for the exact name of package you search for. As in the link here.
Question: I am trying to solve a CTF challenge in which the goal is to guess the generated number. Since the number is huge and you only have 10 attempts per number, I don't think you can apply binary search or any kind of algorithm to solve it, and that it has something to do with somehow getting the seed of the random function and being able to generate the next number, but I have no idea on where to start to get the correct seed. Do you have any idea? Here's the code of the challenge: #!/usr/bin/env python3 import signal import os import random TIMEOUT = 300 assert("FLAG" in os.environ) FLAG = os.environ["FLAG"] assert(FLAG.startswith("CCIT{")) assert(FLAG.endswith("}")) def handle(): for i in range(625): print(f"Round {i+1}") guess_count = 10 to_guess = random.getrandbits(32) while True: print("What do you want to do?") print("1. Guess my number") print("2. Give up on this round") print("0. Exit") choice = int(input("> ")) if choice == 0: exit() elif choice == 1: guess = int(input("> ")) if guess == to_guess: print(FLAG) exit() elif guess < to_guess: print("My number is higher!") guess_count -= 1 else: print("My number is lower!") guess_count -= 1 elif choice == 2: print(f"You lost! My number was {to_guess}") break if guess_count == 0: print(f"You lost! My number was {to_guess}") break if __name__ == "__main__": signal.alarm(TIMEOUT) handle() 

Answer: Don't try guessing the first 624 numbers, just give up on them. You're told what they were, feed them into randcrack as shown in its example. Ask it to predict the next 32-bit number and guess that. For a bigger challenge, you could try it without that tool. Here's some insight, "predicting" the next number, i.e., showing how it's computed from the state: import random for _ in range(5): s = random.getstate()[1] y = s[s[-1]] y ^= (y >> 11); y ^= (y << 7) & 0x9d2c5680; y ^= (y << 15) & 0xefc60000; y ^= (y >> 18); print('predict:', y) print('actual: ', random.getrandbits(32)) print()  Sample output: predict: 150999088 actual: 3825261045 predict: 457032747 actual: 457032747 predict: 667801614 actual: 667801614 predict: 3817694986 actual: 3817694986 predict: 816636218 actual: 816636218  First I get the state, which is 624 words of 32 bits and an index. The calculation of y is from here. The first number is wrong because in reality, the first time the more complicated code above it runs. If you can figure out how to inverse those calculations, you could reconstruct the original state from the given first 624 numbers, then set that state with random.setstate, and generate the next number. The tool does seem to do that inversal, but it doesn't look trivial.
Question: for example I have a pandas DataFrame of the test results in some class. It could look like this table:     Name English French History Math Physic Chemistry Biology     Mike 3 3 4 5 6 5 4   Tom 4 4 3 4 4 5 5   Nina 5 6 4 3 3 3 5   Anna 4 3 4 5 5 3 3   Musa 5 5 4 4 4 6 5   Maria 4 3 5 4 3 2 3   Chris 6 5 5 5 5 5 6     For every student I want to create at least two columns with the best test result and best subject. Important: every student can have more than only one best subject (the results are similar)! For the example above it should be look like this:     Name English French History Math Physic Chemistry Biology Best result Best subject 1 Best subject 2     Mike 3 3 4 5 6 5 4 6 Physic None   Tom 4 4 3 4 4 5 5 5 Chemistry Biology   Nina 5 6 4 3 3 3 5 6 French None   Anna 4 3 4 5 5 3 3 5 Math Physic   Musa 5 5 4 4 4 6 5 6 Chemistry None   Maria 4 3 5 4 3 2 3 5 History None   Chris 6 5 5 5 5 5 6 6 English Biology     what is the best way to do it in Pandas? Thank you in advance! what is the best way to do it in Pandas? Thank you in advance!

Answer: Another possible solution : tmp = df.set_index("Name") # a DataFrame bre = tmp.max(axis=1) # a Series bsu = ( ((tmp.columns + "|") @ tmp.eq(bre, axis=0).T) .str.strip("|").str.split("|", expand=True) .rename(lambda x: f"Best subject {x+1}", axis=1) ) out = tmp.assign(**{"Best result": bre}).join(bsu).reset_index()#.fillna("None")  Output :      Name English French History Math Physic Chemistry Biology Best result Best subject 1 Best subject 2     0 Mike 3 3 4 5 6 5 4 6 Physic    1 Tom 4 4 3 4 4 5 5 5 Chemistry Biology   2 Nina 5 6 4 3 3 3 5 6 French    3 Anna 4 3 4 5 5 3 3 5 Math Physic   4 Musa 5 5 4 4 4 6 5 6 Chemistry    5 Maria 4 3 5 4 3 2 3 5 History    6 Chris 6 5 5 5 5 5 6 6 English Biology
Question: I have a big dataframe with columns including ID and multiple values and different rows can have same or different ID values. I would like to create a new dataframe so, that every row has only one ID and the specific column values are just appended next to the ID. The Dataframe also has other columns with additional values that are same for same ID rows that i would like to keep     ID type1 type2 value1 value2 value3     1 dog yellow 1 2 3   1 dog yellow 5 6 7   2 cat brown 1 1 1   3 mouse blue 1 1 1   1 dog yellow 1 2 3     expected output:     ID type1 type2 value     1 dog yellow 1 2 3 5 6 7 1 2 3   2 cat brown 1 1 1   3 mouse blue 1 1 1     I have been exploring the groupby option, can't get it to have this kind of output

Answer: You can melt and groupby.agg: group = ['ID', 'type1', 'type2'] out = df.melt(group).groupby(group, as_index=False)['value'].agg(list)  Output:  ID type1 type2 value 0 1 dog yellow [1, 5, 1, 2, 6, 2, 3, 7, 3] 1 2 cat brown [1, 1, 1] 2 3 mouse blue [1, 1, 1]  If order matters: out = (df.set_index(group).stack().groupby(group).agg(list) .reset_index(name='value') )  Output:  ID type1 type2 value 0 1 dog yellow [1, 2, 3, 5, 6, 7, 1, 2, 3] 1 2 cat brown [1, 1, 1] 2 3 mouse blue [1, 1, 1]
Question: I want to download spacy, but the version of typing-extensions is lowered in the terminal: ERROR: pydantic 2.3.0 has requirement typing-extensions>=4.6.1, but you'll have typing-extensions 4.4.0 which is incompatible. ERROR: pydantic-core 2.6.3 has requirement typing-extensions!=4.7.0,>=4.6.0, but you'll have typing-extensions 4.4.0 which is incompatible. Installing collected packages: typing-extensions Attempting uninstall: typing-extensions Found existing installation: typing-extensions 4.7.1 Uninstalling typing-extensions-4.7.1: Successfully uninstalled typing-extensions-4.7.1 Successfully installed typing-extensions-4.4.0  Next I want to install the language pack python -m spacy download en, but another error occurs： (base) E:\Anaconda>python -m spacy download en Traceback (most recent call last): File "E:\Anaconda\lib\site-packages\confection\__init__.py", line 38, in <module> from pydantic.v1 import BaseModel, Extra, ValidationError, create_model File "E:\Anaconda\lib\site-packages\pydantic\__init__.py", line 13, in <module> from . import dataclasses File "E:\Anaconda\lib\site-packages\pydantic\dataclasses.py", line 11, in <module> from ._internal import _config, _decorators, _typing_extra File "E:\Anaconda\lib\site-packages\pydantic\_internal\_config.py", line 9, in <module> from ..config import ConfigDict, ExtraValues, JsonEncoder, JsonSchemaExtraCallable File "E:\Anaconda\lib\site-packages\pydantic\config.py", line 9, in <module> from .deprecated.config import BaseConfig File "E:\Anaconda\lib\site-packages\pydantic\deprecated\config.py", line 6, in <module> from typing_extensions import Literal, deprecated ImportError: cannot import name 'deprecated' from 'typing_extensions' (E:\Anaconda\lib\site-packages\typing_extensions.py)  My current python version is 3.7, should I update it? Or is there any better solution? I'm a newbie in this area, thank you all！

Answer: You should use typing_extensions==4.7.1 try : pip install typing_extensions==4.7.1 --upgrade  I also suggest you to upgrade your python version from 3.7 to 3.10 or 3.11 See a relevant answer: https://github.com/tiangolo/fastapi/discussions/9808
Question: I have a pandas dataframe as follows: df1 site_id date hour reach maid 0 16002 2023-09-02 21 NaN 33f9fad6-20c5-426c-962f-bc2fbb82aecb 1 16002 2023-09-04 17 NaN 33f9fad6-20c5-426c-962f-bc2fbb82aecb 2 16002 2023-09-04 19 NaN 4a676aeb-6f6f-4622-934b-59b8f149aad7 3 16002 2023-09-04 17 NaN 35363191-c6aa-49fb-beb1-04a98898bed2 4 16002 2023-09-03 22 NaN a44beb20-a90a-4135-be18-6dda71eeb7c2  I have created another dataframe based on the above dataframe that provides the count of records for each [site_id,date,hour] combination. T df2 site_id date hour count 1666 37226 2023-09-02 8 4586 1676 37226 2023-09-03 16 3586 639 36972 2023-09-03 21 235 640 36972 2023-09-03 22 5431 641 36972 2023-09-03 23 343  I want to filter the first data frame and get exact number of records as given in the count column of second data frame. For example, I want to get the 4586 records from the first data frame matching the site_id 37226, date 2023-09-02 and hour 8. I tried using a forloop on the second dataframe as follows: for index,rows in k3.iterrows(): sid=rows['site_id'] dt=rows['date'] hr=rows['hour'] cnt=rows['count'] kdf1=dff[(dff['site_id'] == sid) & (dff['date']==dt) & (dff['hour']==hr)] kdf2=kdf1[:cnt]  This works - but works extremely slow. Is there a faster method to get the subset. I am also attaching herewith the links to both sample dataframes: Link to df1 and df2 

Answer: You can merge the count from df2 to df1, and then using .groupby to reduce the count of groups: cols = ["site_id", "date", "hour"] df1 = df1.merge(df2, on=cols, how="right") df1 = df1.groupby(cols, group_keys=False).apply(lambda x: x[: x["count"].iloc[0]]) df1.pop("count") print(df1.head())  Prints:  site_id date hour reach maid 0 37221 2023-09-03 19 NaN 3e769e74-9129-49ba-838d-c36f3a9b3335 1 37221 2023-09-03 19 NaN 71e258d2-5155-4001-9b3c-02a1a1f9c9fb 2 37221 2023-09-03 19 NaN 92eaee88-b41c-4999-b1b8-6be183e5d2cf 3 37221 2023-09-03 19 NaN c6eb504a-9259-410b-8391-7b06b3e92a41 4 37221 2023-09-03 19 NaN c36400ff-0790-4844-b58b-2e4cdaafb4d9  Note: With your data this method takes ~0.15 seconds, your original version ~11.2 seconds.
Question: I have a few ranges (that can overlap) as parameter; for example: # tuple[0] <= n < tuple[1] ranges = [(70, 80), (80, 120), (120, 130), (120, 2000), (1990, 2000), (2000, 2040), (2040, 2050)]  And I have a list of tuples as input, where the second element of each tuple is the number that determines the range(s) the tuple belongs. tuples = [('a', 71), ('b', 79), ('c', 82), ('d', 121), ('e', 1991), ('f', 2010), ('g', 2045), ('h', 3000)]  I need to determine the members of each range and store the results in the following manner: # ranges[i] => members[i] members = [{71: 'a', 79: 'b'}, {82: 'c'}, {121: 'd'}, {121: 'd', 1991: 'e'}, {1991: 'e'}, {2010: 'f'}, {2045: 'g'}]  My approach is to create a dict that associates a number to the range(s) it belongs: members = [{} for _ in ranges] number_to_members = defaultdict(list) for i,(x0,x1) in enumerate(ranges): for x in range(x0,x1): number_to_members[x].append(members[i]) for c,n in tuples: if n in number_to_members: for m in number_to_members[n]: m[n] = c  Now the problem is that I need to classify tens of millions of different lists of tuples using the same ranges; the current implementation would require to generate number_to_members for each list of tuples, which is a lot of overhead. How do you work around that problem?  UPDATE Adding an example of the actual problem: from collections import defaultdict ranges = [(70, 80), (80, 120), (120, 130), (120, 2000), (1990, 2000), (2000, 2040), (2040, 2050)] inputs = [ [('a', 71), ('b', 79), ('c', 82), ('d', 121), ('e', 1991), ('f', 2010), ('g', 2045), ('h', 3000)], [('x', 75), ('y', 78), ('z', 1995)] ] members = [{} for _ in ranges] number_to_members = defaultdict(list) for i,(x0,x1) in enumerate(ranges): for x in range(x0,x1): number_to_members[x].append(members[i]) for tuples in inputs: for c,n in tuples: if n in number_to_members: for m in number_to_members[n]: m[n] = c print(members)  output: [{71: 'a', 79: 'b'}, {82: 'c'}, {121: 'd'}, {121: 'd', 1991: 'e'}, {1991: 'e'}, {2010: 'f'}, {2045: 'g'}] [{71: 'a', 79: 'b', 75: 'x', 78: 'y'}, {82: 'c'}, {121: 'd'}, {121: 'd', 1991: 'e', 1995: 'z'}, {1991: 'e', 1995: 'z'}, {2010: 'f'}, {2045: 'g'}] 

Answer: Given the sizes you mentioned, your code in UPDATE seems speed-wise rather optimal already, as it takes time proportional to the size of your desired outputs. It's just incorrect, since it accumulates the input list data instead of computing the members list for each input alone. Since you said you don't need the members after each iteration anymore, you can fix the issue by simply clearing the dicts: for tuples in inputs: ... print(members) for m in members: m.clear()  If you change your mind and do need the members lists later, you can make a copy before you clear: members_lists = [] for tuples in inputs: ... print(members) copy = [m.copy() for m in members] members_lists.append(copy) for m in members: m.clear()
Question: I am using plotly express to create different graphs. I am trying to convert graphs into json format to save in json file. While doing so I am getting error using different ways as below: Way-1 code gives error as below Error-2 Object of type ndarray is not JSON serializable Way-2 code gives error as below Error-2 Object of type Figure is not JSON serializable  Below is MWE: import json import dash_bootstrap_components as dbc from dash import dcc from dash_bootstrap_templates import load_figure_template import plotly.express as px import plotly.io as pio import pandas as pd def generate_pie_charts(df, template) -> list[dict[str, Any]]: pie_charts = list() for field in df.columns.tolist(): value_count_df = df[field].value_counts().reset_index() cols = value_count_df.columns.tolist() name: str = cols[0] value: str = cols[1] try: # Way-1 # figure = px.pie( # data_frame=value_count_df, # values=value, # names=name, # title=f"Pie chart of {field}", # template=template, # ).to_plotly_json() # pie_chart = dcc.Graph(figure=figure).to_plotly_json() # Way-2 figure = px.pie( data_frame=value_count_df, values=value, names=name, title=f"Pie chart of {field}", template=template, ) figure = pio.to_json(figure) # figure = pio.to_json(figure).encode() pie_chart = dcc.Graph(figure=pio.from_json(figure)).to_plotly_json() # pie_chart = dcc.Graph(figure=pio.from_json(figure.decode())).to_plotly_json() pie_charts.append(pie_chart) except Exception as e: print(f"Error-1 {e}") print(f"Length {len(pie_charts)}") return pie_charts def perform_exploratory_data_analysis(): rows = list() template = "darkly" load_figure_template(template) info = { "A": [ "a", "a", "b", "b", "c", "a", "a", "b", "b", "c", "a", "a", "b", "b", "c", ], "B": [ "c", "c", "c", "c", "c", "a", "a", "b", "b", "c", "a", "a", "b", "b", "c", ], } df = pd.DataFrame(info) try: row = dbc.Badge( "For Pie Charts", color="info", className="ms-1" ).to_plotly_json() rows.append(row) row = generate_pie_charts(df, template) rows.append(row) data = {"contents": rows} status = False msg = "Error creating EDA graphs." file = "eda.json" with open(file, "w") as json_file: json.dump(data, json_file) msg = "EDA graphs created." status = True except Exception as e: print(f"Error-2 {e}") result = (status, msg) return result perform_exploratory_data_analysis()  What I am missing?

Answer: You were close with way-2, you need to :  Convert the figure (go.Figure) to a JSON string using pio.to_json(), so that ndarray and other python types used in the figure's data are properly converted into their javascript equivalent.  Deserialize the JSON string using json.loads() in order to get the figure as a standard JSON dict and prevent double encoding with future call to json.dump() (nb. pio.from_json() would return a go.Figure which json.dump() doesn't know how to serialize).   The relevant part of the code : # ... figure = px.pie( data_frame=value_count_df, values=value, names=name, title=f"Pie chart of {field}", template=template, ) # serializable figure dict fig_dict = json.loads(figure.to_json()) pie_chart = dcc.Graph(figure=fig_dict).to_plotly_json() # ...
Question: My company blocks the Windows Task Scheduler (along with many other Python libraries), so I need to use the schedule library. My script imports several flat files, performs some grouping and simple calculations using dataframes, and then saves one final TXT file. I've read up on how schedule works, but how do I use it to run the entire script? import pandas as pd import time import schedule def daily(): #then do all dataframe creation/manipulation, group by's, concatenations, etc... schedule.every().day.at("16:30").do(daily)  It doesn't error out, but it also doesn't do anything; it just states:  do daily() (last run: [never], next run: 2023-09-08 16:30:00 

Answer: The do function only registers the job, but it will not actually execute it. You need to use the run_pending function to execute the job. import pandas as pd import time import schedule def daily(): #dataframe creation/manipulation etc... schedule.every().day.at("16:30").do(daily) while True: schedule.run_pending() time.sleep(60)  The interval of 60 seconds should be sufficient for your case since you are working with minute-level time accuracy and not seconds.  run_pending()  Run all jobs that are scheduled to run. Please note that it is intended behavior that run_pending() does not run missed jobs. For example, if you’ve registered a job that should run every minute and you only call run_pending() in one hour increments then your job won’t be run 60 times in between but only once.