eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe. Column one has a list of numbers. Second column has average of list of numbers in column one. I need to create third column such that I subtract mean value from each of the elements of column one. df = pd.DataFrame({'A':[[4.2,2.3,6.5,2.3],[4.1,5.3,6.5,3.8]]}) df['avg'] = df['A'].apply(lambda p: np.average(p)) df['a_avg' = df['A'].apply(lambda p: (np.array(p)-df['avg']).to_list())  Expected output: df A avg a_avg 0 [4.2,2.3,6.5,2.3] 3.825 [0.375, -1.525, 2.675, -1.525] 1 [4.1,5.3,6.5,3.8] 4.925 [-0.825, 0.375, 1.575, -1.125]  I created column two for my clarity. if there is a way we can directly get column three from column one, that is also good. whats wrong with the code i have written?

Answer: Use list comprehension with convert lists to numpy for improve performance: df['a_avg'] = [(np.round(np.array(p) - np.average(p), 3)).tolist() for p in df['A']]  Or: df['a_avg'] = df.A.apply(lambda p: (np.round(np.array(p) - np.average(p), 3)).tolist()) print (df) A a_avg 0 [4.2, 2.3, 6.5, 2.3] [0.375, -1.525, 2.675, -1.525] 1 [4.1, 5.3, 6.5, 3.8] [-0.825, 0.375, 1.575, -1.125]  Vectorized solution working if each list has same length: arr = np.array(df.A.tolist()) df['a_avg'] = np.round(arr - np.average(arr), 3).tolist() print (df) A a_avg 0 [4.2, 2.3, 6.5, 2.3] [-0.175, -2.075, 2.125, -2.075] 1 [4.1, 5.3, 6.5, 3.8] [-0.275, 0.925, 2.125, -0.575]  Testing performance: df = pd.DataFrame({'A':[[4.2,2.3,6.5,2.3],[4.1,5.3,6.5,3.8]]}) #2k rows df = pd.concat([df] * 1000, ignore_index=True) #Timeless solution In [36]: %timeit df["a_avg"] = [[round(e - np.average(lst), 3) for e in lst] for lst in df["A"]] 118 ms ± 659 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   In [37]: %timeit df['a_avg'] = [(np.round(np.array(p) - np.average(p), 3)).tolist() for p in df['A']] 37.1 ms ± 162 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [38]: %timeit df['a_avg'] = df.A.apply(lambda p: (np.round(np.array(p) - np.average(p), 3)).tolist()) 37.7 ms ± 446 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [39]: %%timeit ...: arr = np.array(df.A.tolist()) ...: df['a_avg'] = np.round(arr - np.average(arr), 3).tolist() ...: 1.36 ms ± 46.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  Testing in 20k rows #20k rows df = pd.concat([df] * 10000, ignore_index=True) In [41]: %timeit df["a_avg"] = [[round(e - np.average(lst), 3) for e in lst] for lst in df["A"]] 1.18 s ± 5.82 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In [42]: %timeit df['a_avg'] = [(np.round(np.array(p) - np.average(p), 3)).tolist() for p in df['A']] 366 ms ± 1.25 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In [43]: %timeit df['a_avg'] = df.A.apply(lambda p: (np.round(np.array(p) - np.average(p), 3)).tolist()) 364 ms ± 824 µs per loop (mean ± std. dev. of 7 runs, 1 loop each) In [44]: %%timeit ...: arr = np.array(df.A.tolist()) ...: df['a_avg'] = np.round(arr - np.average(arr), 3).tolist() ...: ...: 13.7 ms ± 111 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Question: I am trying to read in data from Databricks Hive_Metastore with PySpark. In screenshot below, I am trying to read in the table called 'trips' which is located in the database nyctaxi. Typically if this table was located on a AzureSQL server I was use code like the following: df = spark.read.format("jdbc")\ .option("url", jdbcUrl)\ .option("dbtable", tableName)\ .load()  Or if the table was in the ADLS I would use code similar to the following: df = spark.read.csv("adl://mylake.azuredatalakestore.net/tableName.csv",header=True)  Can some let me know how I would read in the table using PySpark from Databricks Database below:  The additional screenshot my also help  Ok, I've just realized that I think I should be asking how to read tables from "samples" meta_store. In any case I would like help reading in the "trips" table from the nyctaxi database please.

Answer: The samples catalog can be accessed in using spark.table("catalog.schema.table"). So you should be able to access the table using: df = spark.table("samples.nyctaxi.trips")  Note also if you are working direct in databricks notebooks, the spark session is already available as spark - no need to get or create.
Question: I have a dictionary as follows: D = {'mark': [['height', 7], ['weight', 70]], 'david': [['height', 8], ['weight', 80]], 'john': [['height', 9], ['weight', 90]]} print (D)  I wanted to get the pandas dataframe into this form:  names height weight 0 mark 7 70 1 david 8 80 2 john 9 90  I tried as follows. df = pd.DataFrame(D) print (df)  However, it produces a wrong result below.  mark david john 0 [height, 7] [height, 8] [height, 9] 1 [weight, 70] [weight, 80] [weight, 90]  Edit: What if dictionary has not same number of values as follows? D2 = {'mark': [['height', 7], ['weight', 70]], 'david': [['height', 8], ['weight', 80]], 'john': [['height', 9]]} print (D2) 

Answer: You can use : df = pd.DataFrame({k: dict(v) for (k,v) in D.items()}).T.reset_index(names="names")  Output : print(df) names height weight 0 mark 7 70 1 david 8 80 2 john 9 90
Question: When working in a Git repository where my Python/Django source is in a subfolder {$workspace}/app, as seen below.  project/  .vscode/ .git/ app/ -- the app source code (not a Python module) docs/ .gitignore LICENSE    The problem is that VSCode adds an incorrect app. prefix when auto-generating import statements. For example, it will generate app.core when the actual project module is core. Also, there isn't an __init__.py in the /app directory. I have read that I can open a workspace in the app/ directory, but the project repository is one directory above the app source and I sometimes need to make changes at the project root level. I have tried the following items in settings.json, which don't seem correct when all I want to do is configure the primary auto-complete and analysis paths. { "python.analysis.extraPaths": [ "/app", ], "python.autoComplete.extraPaths": [ "/app", ], {  How can I get VSCode to stop treating the project /app directory as a Python module?

Answer: If it's the file structure in your question, open the project as a workspace. settings.json should be like this { "python.analysis.extraPaths": [ "./app", ], "python.autoComplete.extraPaths": [ "./app", ], }
Question: I'm trying to make my 'hacking' portion of a Fallout terminal simulator more akin to what is actually in the game. I'm in the process of setting it up so that it uses a word list, picks some words from it, and then puts them into the list of address-like display. However, because the length of the words ranges from 4 letters (FISH) to 12 letters (Hippopotamus), the lines get staggered, and the pipes I'm using to separate the options no longer line up. I tried using an answer I found here, which works fine for ust spaces but trying to insert {"+varNameHere+":<15}".format(a)  in the middle of the string, (typing_.typingPrint('0xCC01 |{)&@*'+"{:"+fillerChar+"^12}".format(words[0])+'@ |0xCC02 |!(#'+"{:"+fillerChar+"^12}".format(words[0])+'@)!#\n')  , throws an ValueError about there only being one '}' in the format string. I understand what the error is talking about, but the reason for my question is, Is there anyway to have the padding character be randomly decided when the string is printed out into the terminal window? Code: possibleChars = ['!','@','#','$','%','^','&','*','(',')',';',':'] fillerChar = random.sample(possibleChars) print('0xCC01 |{)&@*'+"{:"+fillerChar+"^12}".format(words[0])+'@ |0xCC02 |!(#'+"{:"+fillerChar+"^12}".format(words[0])+'@)!#\n') 

Answer: Create a random 15-character string. Then extract a slice of it long enough to pad out what you want. random_string = 'weopi94nf0683d0' word = 'FISH' left = (len(random_string) - len(word)) // 2 right = left + len(word) padded_word = random_string[:left] + word + random_string[right:]
Question: Using a microphone as an input for real-time audio. How do I extract the currently said phoneme from the audio? I need it for lipsyncing 2d characters. Basically, my approach would be to:  Fetch the real-time audio using a microphone Detect the current phoneme that is being pronounced from the audio.  I have tried looking everywhere for an example or library that could solve this type of problem. Most libraries don't seem to output phonemes from audio. There is a website that explains how they used machine learning to solve this, however without any code or tutorial on how to do it. https://www.arxiv-vanity.com/papers/1910.08685/ There is also this cool speech recognition tool called Pocketsphinx, but I cannot seem to find an example of it using Phoneme Recognition yet.

Answer: The way I would approach this is to get the word from the audio using Whisper or a similar STT service (the Python Speech Recognition Library is the go-to at the moment), then I would use the CMU Dict Library to provide phonemes for each word. The phonemes are given using the CMU dictionary - for example DH for the θ phoneme - the th sound in this and that. That is, they are not given in IPA pronunciation - so you may need another layer if you need the phonemes in IPA format. If you need IPA formatted phonemes, then consider the IPA2 library.
Question: I have started playing with Data Analysis and all the related tools: Pandas, Numpy, Jupyter etc... The task I am working on is simple, and I could do easily with regular python. However I am more interested in exploring Pandas, and I am looking therefore for a Pandas solution. I have this simple Pandas DataFrame. The timestamp column is just a Unix timestamp, but to make things more readable I have just put a more comfortable number:     id timestamp success     1 9999 True   2 1111 True   3 9999 False   4 1111 True   5 9999 True   6 1111 True     I want to group by timestamp, but I want another aggregate column which is the result of the success column: if True count as one, if False count as 0. I hope the table below clarify what I am trying to achieve. Basically 1111 has three True therefore the sum is 3. 9999 has two True and one False, therefore the sum is 2.     timestamp success     1111 3   9999 2    

Answer: import pandas as pd # The DataFrame a = { 'id': [1, 2, 3, 4, 5, 6], 'timestamp': [9999, 1111, 9999, 1111, 9999, 1111], 'success': [True, True, False, True, True, True] } df = pd.DataFrame(a) # Group by timestamp and calculate the sum of success result = df.groupby('timestamp')['success'].sum().reset_index() # Result print(result)  Do you mean this? You group your data frame by timestamp and then you find the frequency of true values.
Question: I am trying to show a light-gray image using plt.imshow(), but the image turns out black. I tried: import matplotlib.pyplot as plt import numpy as np test_image = np.zeros((3871, 2484)) test_image.fill(200) plt.imshow(test_image, cmap="gray") plt.show()  But ended up getting:   Matplotlib version: 3.7.1 Numpy version: 1.24.3 Python version: 3.11.3 

Answer: You have to include the vmin,vmax parameters when you plot a single color image with plt.imshow(...). Set vmin=0 and vmax=500 to get a gray image. If vmin,vmax are not specified, then they will be set to the min and max values of the image data. This means that all of your input data is equal to vmin, which is the darkest possible value (black). import matplotlib.pyplot as plt import numpy as np test_image = np.zeros((3871, 2484)) test_image.fill(200) plt.imshow(test_image, cmap="gray", vmin=0, vmax=500) plt.show()
Question: I am attempting to visualize some data as a table, where the boxes of each table element are colored according to their value, the numerical value is also displayed, and the uncertainty on each element is shown. I can achieve 2 out of these 3 things using pandas.pivot_table and sns.heatmap, but cannot seem to include the uncertainty on each table element as part of the annotation. In the example code snippet: import pandas as pd import seaborn as sns import numpy as np df = pd.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo", "bar", "bar", "bar", "bar"], "B": ["one", "one", "one", "two", "two", "one", "one", "two", "two"], "C": ["small", "large", "large", "small", "small", "large", "small", "small", "large"], "D": [1, 2, 2, 3, 3, 4, 5, 6, 7], "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]}) table = pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'], aggfunc=np.sum, fill_value=0) sns.heatmap(table,annot=True)  we produce a table like so:  However, imagine that the entries "E" represented the uncertainty on elements "D". Is there any way these can be displayed on the table, as "E"[i]+/-"D"[i]? I tried using a custom annotation grid, but this requires a numpy array and so string formatting each element didn't work for this.

Answer: You can pass a DataFrame with the formatted strings to sns.heatmap: table = pd.pivot_table(df, values=['D', 'E'], index=['A', 'B'], columns=['C'], aggfunc=np.sum, fill_value=0) sns.heatmap(table['D'], annot=table['D'].astype(str)+'±'+table['E'].astype(str), fmt='')
Question: what is the fastest way to remove an element from a list by its value? I believe list.remove("element_to_be_removed") is the naive way. How can we optimize it?

Answer: Finding an element in a list by value is O(n), as is removing it once you have found it. There is no way to reduce this; it's inherent in how lists are built. Finding and/or removing an element in a set is O(1). Converting a list into a set is O(n). If you have a list and you need to remove one item, converting it to a set and then removing one item doesn't get you to O(1), because the set conversion itself is an O(n) operation. You're better off using list.remove(). If you have a list of unique items, and you anticipate needing to remove many items from it in an unordered way (say, you're going to eventually remove each item one by one), you can change that from an O(n^2) operation to an O(n) operation by converting the entire list into a set (once, in O(n)) and then removing the elements one by one after that (in O(1) per item, hence O(n) overall). The ideal solution (if possible) is to anticipate the ways in which you'll need to use this collection of items and make it a set from the beginning if the functionality of a set is a better fit for what you're doing.
Question: I would like to extract the values of a dictionary inside a Pandas DataFrame df into new columns of that DataFrame. All keys in the referring dict are the same across all rows. import pandas as pd df = pd.DataFrame({'a': [1, 2, 3], 'b': [{'x':[101], 'y': [102], 'z': [103]}, {'x':[201], 'y': [202], 'z': [203]}, {'x':[301], 'y': [302], 'z': [303]}]})   dfResult = pd.DataFrame({'a': [1, 2, 3], 'x':[101, 201, 301], 'y': [102, 202, 302], 'z': [103, 203, 303]})   I am as far as I can get the keys and values out of the dict, but I do not know how to make new columns out of them: df.b.apply(lambda x: [x[y] for y in x.keys()]) 0 [[101], [102], [103]] 1 [[201], [202], [203]] 2 [[301], [302], [303]] df.b.apply(lambda x: [y for y in x.keys()]) 0 [x, y, z] 1 [x, y, z] 2 [x, y, z] 

Answer: If there are always one element lists is possible use nested list with dictionary comprehension and pass to DataFrame constructor: df = df.join(pd.DataFrame([{k: v[0] for k, v in x.items()} for x in df.pop('b')], index=df.index)) print (df) a x y z 0 1 101 102 103 1 2 201 202 203 2 3 301 302 303  Another idea is create DataFrame for each row in dictionary comprehension and join by concat: df = df.join(pd.concat({k: pd.DataFrame(v) for k, v in df.pop('b').items()}).droplevel(1)) print (df) a x y z 0 1 101 102 103 1 2 201 202 203 2 3 301 302 303
Question: When grouping a Polars dataframe in Python, how do you concatenate string values from a single column across rows within each group? For example, given the following DataFrame: import polars as pl df = pl.DataFrame( { "col1": ["a", "b", "a", "b", "c"], "col2": ["val1", "val2", "val1", "val3", "val3"] } )  Original df: shape: (5, 2) ┌──────┬──────┐ │ col1 ┆ col2 │ │ --- ┆ --- │ │ str ┆ str │ ╞══════╪══════╡ │ a ┆ val1 │ │ b ┆ val2 │ │ a ┆ val1 │ │ b ┆ val3 │ │ c ┆ val3 │ └──────┴──────┘  I want to run a group_by operation, like:  df.group_by('col1').agg( col2_g = pl.col('col2').some_function_like_join(',') )  The expected output is: ┌──────┬───────────┐ │ col1 ┆ col2_g │ │ --- ┆ --- │ │ str ┆ str │ ╞══════╪═══════════╡ │ a ┆ val1,val1 │ │ b ┆ val2,val3 │ │ c ┆ val3 │ └──────┴───────────┘  What is the name of the some_function_like_join function? I have tried the following methods, and none work: df.group_by('col1').agg(pl.col('col2').list.concat(',')) df.group_by('col1').agg(pl.col('col2').join(',')) df.group_by('col1').agg(pl.col('col2').list.join(',')) 

Answer: If you want to concatenate them, I assume you want the result as a string with your specified delimiter: out = df.group_by("col1").agg( pl.col("col2").str.join(",") )  Result: shape: (3, 2) ┌──────┬───────────┐ │ col1 ┆ col2 │ │ --- ┆ --- │ │ str ┆ str │ ╞══════╪═══════════╡ │ a ┆ val1,val1 │ │ b ┆ val2,val3 │ │ c ┆ val3 │ └──────┴───────────┘  If you want them within a List, you simply do: out = df.groupby("col1").agg( pl.col("col2") )  Result: shape: (3, 2) ┌──────┬──────────────────┐ │ col1 ┆ col2 │ │ --- ┆ --- │ │ str ┆ list[str] │ ╞══════╪══════════════════╡ │ a ┆ ["val1", "val1"] │ │ c ┆ ["val3"] │ │ b ┆ ["val2", "val3"] │ └──────┴──────────────────┘
Question: I have a list of tasks that each take a different amount of time. Let's say, I have 3 tasks, with durations close to 1x, 5x, 10*x. My tqdm code is something like: from tqdm import tqdm def create_task(n): def fib(x): if x == 1 or x == 0: return 1 return fib(x - 1) + fib(x - 2) return lambda: fib(n) n = 1 tasks = [create_task(n), create_task(5*n), create_task(10*n)] for task in tqdm(tasks): task.run()  The problem is that tqdm thinks each iteration takes the same amount of time. As the first takes approximately 1/10 of the time, the ETA is unreliable. My question: is it possible to somehow add a hint to tqdm to inform how much each iteration takes compared to the first? Something like informing the duration weights of each iteration... Thanks!

Answer: There are two standard usages for tqdm progress bars: iterable-based, and manual. Manually updating a progress bar allows you to specify progress bar weights. Consider the following code: def my_func(x): """ Sleep for x / 5 seconds. """ duration = x / 5 time.sleep(duration) in_vals = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] # Ex. 1: unweighted, updated automatically (iterable-based) for x in tqdm(in_vals, position=0): my_func(x) # Ex. 2: unweighted, updated manually # functionally identical to example 1 with tqdm(total=len(in_vals), position=0) as pbar: for x in in_vals: my_func(x) pbar.update(n=1) # Ex. 3: weighted, updated manually with tqdm(total=sum(in_vals), position=0) as pbar: for x in in_vals: my_func(x) pbar.update(n=x)  The first two examples give a progress bar with an inaccurate estimated duration that grows longer as the iterations progress. This is because tqdm doesn't know that later iterations will take longer to complete than earlier iterations. To fix this problem and get an accurate ETA from the start, we specify the total argument in tqdm() as the sum of all weights (example 3). To update the progress bar after completing an iteration with weight w, call pbar.update(n=w). For the OP's use case, a tqdm progress bar with duration weights would look something like this: def fib(x): time.sleep(.01) # added this so that progress bar isn't completed instantly if x == 1 or x == 0: return 1 return fib(x - 1) + fib(x - 2) in_vals = [1, 5, 10] with tqdm(total=sum(in_vals), unit='calc', position=0) as pbar: for num in in_vals: fib(num) pbar.update(num)
Question: I was wandering through the docs of pygbag, and I couldn't find how the python scripts are actually executed from the browser. I made a test project to look how the files created by pygbag looked like, but I couldn't really figure out what role the index.html exactly plays. It seemed to me like I couldn't find any script in it, so I supposed that it could be directly interpreted, but I'm not really sure. There is a python script in the html file, and I found one line which seems to run the main program : await shell.runpy(main, callback=ui_callback), but I don't know whether it just executes the python script in the folder or if the script is somewhere compiled in this file. Could anyone explain me ?

Answer: When running in the webpage pygbag is in fact a C runtime linked to libpython ( cpython-wasm from python.org) compiled to WebAssembly with emscripten compiler and hosted on a CDN (pygame-web.github.io). It is downloaded once per game and per version update for fast local use. There's some javascript glue to connect the C library used by pygbag and python to some file descriptors. You cannot guess the mechanism that easily because calls originate from wasm cpu which is not exposed in javascript console. Those file descriptors manipulated by the libc (musl provided by emsdk the portable emscripten compiler) can be in a virtual filesystem hosted by MEMFS from emscripten runtime ( eg for /tmp ) or BrowserFS a more advanced virtual filesystem ( /data and /usr ). They can also be stdin/stdout/stderr file descriptors and this is why you cannot find the file on startup : the python part of html file is sent to python interpreter as if you typed it in your shell this is done by calling PyRun_InteractiveLoop on that file descriptor. Later if a file "main.py" is found in the Virtual filesystem it is queued but you can also pass relative file url on the command line eg https://pygame-web.github.io/showroom/pypad.html#src/test_panda3d_cube.py. It also work with github gist raw links. pygbag can also embed code or git repo eg https://pygame-web.github.io/showroom/test_embed_git.html directly in html pages. Some packages like pygame-ce, Panda3D or Harfang3D are indeed pre-compiled to WebAssembly and that's because they are mostly C or C++. Python code is actually interpreted and type-annotated code could be compiled direcly to Wasm but that fonctionnality is not (yet) available for public use. The format choosen for game archive is a zip file similar to android APK though unaligned and unsigned. The android runtime to make these run on real android is not (yet) available for public use either. Everyone is welcome to add more informations on pygame-web.github.io wiki.
Question: Here is the code block which caused the error training_args = TrainingArguments( output_dir="my_awesome_mind_model", evaluation_strategy="epoch", save_strategy="epoch", learning_rate=3e-5, per_device_train_batch_size=32, gradient_accumulation_steps=4, per_device_eval_batch_size=32, num_train_epochs=10, warmup_ratio=0.1, logging_steps=10, load_best_model_at_end=True, metric_for_best_model="accuracy", push_to_hub=True, ) trainer = Trainer( model=model, args=training_args, train_dataset=dataset["train"], # eval_dataset=encoded_minds["test"], tokenizer=feature_extractor, compute_metrics=compute_metrics, ) trainer.train()  getting the following error  NameError Traceback (most recent call last) in <cell line: 1>() 1 training_args = TrainingArguments( 2 output_dir="my_awesome_mind_model", 3 evaluation_strategy="epoch", 4 save_strategy="epoch", 5 learning_rate=3e-5, 4 frames /usr/local/lib/python3.10/dist-packages/transformers/training_args.py in _setup_devices(self) 1629 self._n_gpu = 1 1630 else: 1631 self.distributed_state = PartialState(backend=self.ddp_backend) 1632 self._n_gpu = 1 1633 if not is_sagemaker_mp_enabled(): NameError: name 'PartialState' is not defined  I am trying to follow the audio classification guide of hugging face(link on another dataset but upon running the training args code i am getting name "PartialState" not defined error.

Answer: As of 2023-05-11: The error seems to be caused by an issue in the huggingface/accelerate library. You can try following solutions: Reinstall transformers & accelerate pip uninstall -y transformers accelerate pip install transformers accelerate  If you are using colab/Jupyter, make sure to restart the notebook's Runtime. Install dev version of accelerate pip install git+https://github.com/huggingface/accelerate  Reverse to previous version of transformers (4.28.0) # You might also need to uninstall transformers first: pip uninstall -y transformers pip install transformers==4.28.0