eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using Pandas 2.1.3. I am trying to join two DataFrames on multiple index levels, and one of the index levels has NA's. The minimum reproducible example looks something like this: a = pd.DataFrame({ 'idx_a':['A', 'A', 'B'], 'idx_b':['alpha', 'beta', 'gamma'], 'idx_c': [1.0, 1.0, 1.0], 'x':[10, 20, 30] }).set_index(['idx_a', 'idx_b', 'idx_c']) b = pd.DataFrame({ 'idx_b':['gamma', 'delta', 'epsilon', np.nan, np.nan], 'idx_c': [1.0, 1.0, 1.0, 1.0, 1.0], 'y':[100, 200, 300, 400, 500] }).set_index(['idx_b', 'idx_c']) c = a.join( b, how='inner', on=['idx_b', 'idx_c'] ) print(a) x idx_a idx_b idx_c A alpha 1.0 10 beta 1.0 20 B gamma 1.0 30 print(b) y idx_b idx_c gamma 1.0 100 delta 1.0 200 epsilon 1.0 300 NaN 1.0 400 1.0 500 print(c) x y idx_a idx_b idx_c B gamma 1.0 30 100 1.0 30 400 1.0 30 500  I would have expected: print(c) x y idx_a idx_b idx_c B gamma 1.0 30 100  Why is join matching on the NaN values?

Answer: You can resolve your problem by removing the indexes and using merge instead of join: a = pd.DataFrame({ 'idx_a':['A', 'A', 'B'], 'idx_b':['alpha', 'beta', 'gamma'], 'idx_c': [1.0, 1.0, 1.0], 'x':[10, 20, 30] }) b = pd.DataFrame({ 'idx_b':['gamma', 'delta', 'epsilon', np.nan, np.nan], 'idx_c': [1.0, 1.0, 1.0, 1.0, 1.0], 'y':[100, 200, 300, 400, 500] }) c = a.merge(b, on=['idx_b', 'idx_c'], how='inner')  Output:  idx_a idx_b idx_c x y 0 B gamma 1.0 30 100  If you want to keep the indexes on a and b as they are in the question you can do this (thanks @mozway): c = (a .reset_index() .merge(b.reset_index(), on=['idx_b', 'idx_c'], how='inner') .set_index(list(dict.fromkeys(a.index.names+b.index.names))) )  Output:  x y idx_a idx_b idx_c B gamma 1.0 30 100
Question: I have a data frame and I want to remove some rows if their value is not equal to some values that I have stored in a list. So I have a list variable stating the values of objects I want to keep: allowed_values = ["value1", "value2", "value3"]  And I am attempting to remove rows from my dataframe if a certain column does not contain 1 of the allowed_values. At first I was using a for loop and if statement like this: for index, row in df.iterrows(): if row["Type"] not in allowed_values: # drop the row, was about to find out how to do this, but then I found out about the `.loc()` method and thought it would be better to use this.  So using the .loc() method, I can do something like this to only keep objects that have a Type value equal to value1: df = df.loc[df["Type"] == "value1"]  But I want to keep all objects that have a Type in the allowed_values list. I tried to do this: df = df.loc[df["Type"] in allowed_values]  but this gives me the following error: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().  I would expect this to still work as using the in or a combination of not in operators still results in a boolean, so I'm not sure why the .loc() method doesn't like these operators. What exactly is wrong with using in or not operators in the .loc() method and how can I create a logical statment that will drop rows if their Type value is not in the allowed_values list? EDIT: I found this question asking about the same error I got and the answer was that you need to use bitwise operators only (e.g. ==, !=, &, |, etc) and not and in are not bitwise operators and require something called "truth-values". So I think the only way to get the functionality I want is to just have a lengthy bitwise logical operator, something like: df = df.loc[(df["Type"] == "value1") | (df["Type"] == "value2") | (df["Type"] == "value3")]  Is there no other way to check each value is in the allowed_values list? This would make my code a lot neater (I have more than 3 values in the list, so this is a lengthy line).

Answer: Try this: import pandas as pd allowed_values = ['White', 'Green', 'Red'] df = pd.DataFrame({'color': ['White', 'Black', 'Green', 'White']}) df = df[df['color'].isin(allowed_values)] df color 0 White 2 Green 3 White  If you must use .loc then you can use: df = df.loc[df['color'].isin(allowed_values)]
Question: I have a pandas data frame and would like to stack 4 columns to 2. So I have this df = pd.DataFrame({'date':['2023-12-01', '2023-12-05', '2023-12-07'], 'other_col':['a', 'b', 'c'], 'right_count':[4,7,9], 'right_sum':[2,3,5], 'left_count':[1,8,5], 'left_sum':[0,8,4]})   date other_col right_count right_sum left_count left_sum 0 2023-12-01 a 4 2 1 0 1 2023-12-05 b 7 3 8 8 2 2023-12-07 c 9 5 5 4  and would like to get this  date other_col side count sum 0 2023-12-01 a right 4 2 1 2023-12-05 b right 7 3 2 2023-12-07 c right 9 5 3 2023-12-01 a left 1 0 4 2023-12-05 b left 8 8 5 2023-12-07 c left 5 4 

Answer: You can use a custom reshaping with a temporary MultiIndex: out = (df .set_index(['date', 'other_col']) .pipe(lambda x: x.set_axis(x.columns.str.split('_', expand=True), axis=1)) .rename_axis(columns=['side', None]) .stack('side').reset_index() )  Or a melt+pivot: tmp = df.melt(['date', 'other_col'], var_name='side') tmp[['side', 'col']] = tmp['side'].str.split('_', n=1, expand=True) out = (tmp.pivot(index=['date', 'other_col', 'side'], columns='col', values='value') .reset_index().rename_axis(columns=None) )  Output:  date other_col side count sum 0 2023-12-01 a left 1 0 1 2023-12-01 a right 4 2 2 2023-12-05 b left 8 8 3 2023-12-05 b right 7 3 4 2023-12-07 c left 5 4 5 2023-12-07 c right 9 5  Or, much easier, using the janitor library and pivot_longer: # pip install pyjanitor import janitor out = df.pivot_longer(index=['date', 'other_col'], names_to=('side', '.value'), names_pattern=r'([^_]+)_([^_]+)')  Output:  date other_col side count sum 0 2023-12-01 a right 4 2 1 2023-12-05 b right 7 3 2 2023-12-07 c right 9 5 3 2023-12-01 a left 1 0 4 2023-12-05 b left 8 8 5 2023-12-07 c left 5 4
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame( { 'a': [10, 20, 30, 400, 50, 60], 'b': [897, 9, 33, 4, 55, 65] } )  And this is the output that I want. I want to create column c.  a b c 0 10 897 NaN 1 20 9 897.0 2 30 33 NaN 3 400 4 400.0 4 50 55 NaN 5 60 65 NaN  These are the steps needed: a) Find rows that df.a > df.b b) From the above rows compare the value from a to its previous value from b. If it was more than previous b value, put a in column c otherwise put the previous b. For example: a) Rows 1 and 3 met df.a > df.b b) From row 1, 20 is less than 897 so 897 is chosen. However in row 3, 400 is greater than 33 so it is selected. This image clarifies the point:  This is what I have tried but it does not work: df.loc[df.a > df.b, 'c'] = max(df.a, df.b.shift(1)) 

Answer: Try: mask = df.a > df.b df.loc[mask, "c"] = np.where(df["a"] > df["b"].shift(), df["a"], df["b"].shift())[mask] print(df)  Prints:  a b c 0 10 897 NaN 1 20 9 897.0 2 30 33 NaN 3 400 4 400.0 4 50 55 NaN 5 60 65 NaN
Question: -------------------- original question --------------------------------- How to install LightGBM?? I have checked multiple sources but staill failed to install. I tried pip and conda but both return the error: [LightGBM] [Warning] Using sparse features with CUDA is currently not supported. [LightGBM] [Fatal] CUDA Tree Learner was not enabled in this build. Please recompile with CMake option -DUSE_CUDA=1  What i have tried is following: git clone --recursive https://github.com/microsoft/LightGBM cd LightGBM/ mkdir -p build cd build cmake -DUSE_GPU=1 .. make -j$(nproc) cd ../python-package pip install .  -------------------- My solution below (cuda) --------------------------------- Thanks for the replies guys. I tried some ways and finally it works as below: First, make sure cmake is installed (under the wsl): sudo apt-get update sudo apt-get install cmake sudo apt-get install g++  Then, git clone --recursive https://github.com/microsoft/LightGBM cd LightGBM mkdir build cd build cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. make -j4  Currently, the install is not linked to any conda env yet. So to do this, under the vscode terminal (or still wsl), conda activate an env and then create a jupyter notebook for testing: Make sure that lib_lightgbm.so is under the LightGBM/python-package, if not, copy into that folder. Then in the jupyter notebook: import sys import numpy as np sys.path.append('/mnt/d/lgm-test2/LightGBM/python-package') import lightgbm as lgb  The final bit is you can refer Jame's reply that device needs to be set to 'cuda' instead of 'gpu'.

Answer: Seeing logs about CUDA in the original posts suggests to me that you're trying to use CUDA-enabled LightGBM. It's important to clarify that, as LightGBM supports two different GPU-accelerated builds:  -DUSE_GPU=1 ("device": "gpu") = OpenCL-based build targeting a wide range of GPUs -DUSE_CUDA=1 ("device": "cuda") = CUDA kernels targeting NVIDIA GPUs  As described in the project's docs (link), as of v4.0.0 building the lightgbm Python package from sources in its git repos requires use of a shell script in that repo. Run the following to build and install a CUDA-enabled version of the library from the source code on GitHub. git clone --recursive https://github.com/microsoft/LightGBM cd LightGBM/ sh build-python.sh install --cuda  If you'd prefer to install from a release on PyPI without having to clone the repo, run the following. pip install \ --no-binary lightgbm \ --config-settings=cmake.define.USE_CUDA=ON \ 'lightgbm>=4.0.0'  With CUDA-enabled LightGBM installed that way, you can then use GPU-accelerated training by passing "device": "cuda" through parameters, like this: import lightgbm as lgb from sklearn.datasets import make_regression X, y = make_regression(n_samples=10_000) dtrain = lgb.Dataset(X, label=y) bst = lgb.train( params={ "objective": "regression", "device": "cuda", "verbose": 1 }, train_set=dtrain, num_boost_round=5 )
Question: In ODOO 15 I have made my own template that runs a method that returns some data to show in a dictionary. This is a piece of the template: <t t-set="PRI_par_DSP_par_stage" t-value="o.PRI_par_DSP_par_stage(o.date_start, o.date_end, o.source, o.domain)"/> <table border="1" style="text-align: left; width: auto; margin: 0 auto;"> <tbody> <t t-foreach="PRI_par_DSP_par_stage" t-as="row"> <tr> <td><t t-esc="row"/></td> <td><t t-esc="row['dsp_id']"/></td> <td><t t-esc="row['Brouillon']"/></td> [....] </tbody> </table> </t>  And my method returns something like: [{'dsp_id': 'DEBIT', 'Brouillon': 3936.0, 'Qualification': 40299.24, 'Closing': 156753.59}, {'dsp_id': 'THD', 'Closing': 22487.8}]  When rendering this I got an 500 error: Web Error message: Error when render the template KeyError: 'Brouillon' Template: 1026 Path: /t/t/div/main/t/div/div[7]/table/tbody/t/tr/td[3]/t Node: <t t-esc="row['Brouillon']"/> The error occurred while displaying the model and evaluated the following expressions: 1026<t t-esc="row['Brouillon']"/>  Because, of course, I don't have that key in second dictionary from list. Is there a way (other than add those keys in my response) to overcome this issue, like checking if there is really a key and after that trying to render it, I have tried with t-if=row..., t-if=define(row...) still have this issue. Any ideas will be highly appreciated. :)

Answer: <td t-if="'dsp_id' in row.keys()"><t t-esc="row['dsp_id']"/></td> <td t-if="'Brouillon' in row.keys()"><t t-esc="row['Brouillon']"/></td>
Question: I use Python in Power BI and have currently the following code: import matplotlib.pyplot as plt plt.hist(dataset.age, bins=10, edgecolor="#6A9662", color="#DDFFDD", alpha=0.75) plt.show()  In my table i have a column TYPE with "E" and "G". How can i add a Filter TYPE = "E" to the Python code? Many thanks in advance! I am expecting that my histogram shows only results for the filtered table TYPE = "E".

Answer: I suggest you to filter the dataframe before plotting import matplotlib.pyplot as plt data = dataset[dataset["TYPE"]=="E"].age plt.hist(data, bins=10, edgecolor="#6A9662", color="#DDFFDD", alpha=0.75) plt.show()
Question: I am bit rusty in pandas to json transforms. I have a pandas data frame like this for a fictional library:     userId visitId readingTime Books BookType     u1 1 300 book1,book2,book3 Fiction   u2 1 400 book4,book5 Horror   u2 2 250 book6 Romance     Need to create a json that is like: { "visitSummary": { "u1": [ { "readingTime": 300, "Books": [ "book1", "book2", "book3" ], "BookType": "Fiction" } ], "u2": [ { "readingTime": 400, "Books": [ "book4", "book5" ], "BookType": "Horror" }, { "readingTime": 250, "Books": [ "book6" ], "BookType": "Romance" } ] } }  I was thinking to do it using nested loops and processing each row. I am hoping, there is a simpler pythonic way to do it. Using Python 3.10 and pandas 2.1.4

Answer: With split, groupby & to_dict : out = { "visitSummary": (df.assign(Books=df["Books"].str.split(",")) .groupby("userId").apply(lambda g: g.drop( columns=["userId", "visitId"]).to_dict("records")).to_dict()) }  Output : import json; print(json.dumps(out, indent=4)) { "visitSummary": { "u1": [ { "readingTime": 300, "Books": [ "book1", "book2", "book3" ], "BookType": "Fiction" } ], "u2": [ { "readingTime": 400, "Books": [ "book4", "book5" ], "BookType": "Horror" }, { "readingTime": 250, "Books": [ "book6" ], "BookType": "Romance" } ] } }
Question: I am participating in a project using RTSP and cameras footage. My objective is to create a continuous flow of videos streamed in the same address and route to update the videos according to the footage we need to display. So, the expected behavior I have to create is to change the video streamed in the address while using ffplay to visualize the stream I am creating. This is recreated by this diagram:  But this simply does not work. Whenever I try to change the video, the original video just freeze and it is impossible to visualize anything. ffplay throws an error: rtsp://localhost:8554/sample: error while seeking. And this is weird for me because when I shut down the transmition and I open it again, it shows the changed footage that I wanted to visualize now. To create the RTSP server I am using this repository: https://github.com/p513817/rtsp4k. It is based on mediamtx and Python to recreate all the logic. I have no knowledge in RTSP, so I don't know if I should do anything low-level to keep the flow without finishing the video or I should try other framework. I need help! The logic I am doing to update the video is  def put_placeholder(self, input:str, route: str): success = True try: # I change the Displayer to show my placeholder instead the original video self.dprs[route] = Displayer( input='./utils/placeholder.mp4', route=str(route), start_stream=True ) # Same here if route in PARAMS.CONF["streams"] : PARAMS.CONF["streams"][route] = { "input": './utils/placeholder.mp4' } write_config(PARAMS.CONF_PATH, PARAMS.CONF) except Exception as e: logging.exception(e) success = False finally: # I update the info in the .json that summarize all the streams self._update_info( input='./utils/placeholder.mp4', route=route, url=self.dprs[route].get_url() if success else "", status=success ) return self.info[route] 

Answer: I realized that I was creating a new object instead of changing the current one. This made that ffplay was incapable of keeping track of the address and route because loses connection and then it is reconnected immediately, making the stream to crash. The final code for updating the videos is a modification of the function put_placeholder that I created in the Manager class in rtsp_handler.py. It is added a new function called change_input() in the Displayer class by creating a function that updates the file's path and not creates a new thread or class instance.  def put_placeholder(self, input:str, route: str): success = True try: # NOTE: modify config path = "" if input == 'placeholder': path = './utils/placeholder.mp4' else: path = './data/'+input+'.mp4' self.dprs[route].change_input(path) if route in PARAMS.CONF["streams"] : PARAMS.CONF["streams"][route] = { "input": path } write_config(PARAMS.CONF_PATH, PARAMS.CONF) except Exception as e: logging.exception(e) success = False finally: self._update_info( input=path, route=route, url=self.dprs[route].get_url() if success else "", status=success ) return self.info[route]   def change_input(self, new_input:str): self.input = new_input self.src = self._create_source()
Question: I'm cleaning up a dataset for plotting. Pretty standard stuff, transposing, deleting unnecessary columns, etc. Here is the code so far: #File name fname = r'E:\Grad School\Research\BOEM\Data\Grain Size\ForScript\allmyCoresRunTD.csv' #Read csv file into a dataframe. skips the first row that just has the 'record number'. Also only loads the #Dx rows and not the remainder of the bins. df = pd.read_csv(fname, index_col=None, skiprows=1, nrows=8) #Delete 'Average' from Malvern. The script will average the data for us df.drop(columns = df.columns[df.columns.str.startswith('Average')], inplace=True) df.drop(columns = df.columns[df.columns.str.startswith('FSH')], inplace=True)  This is what the dataframe looks like at this point:  I rearrange the data some: #Transpose the data df2 = df.transpose() #Change the first row into column names df2.columns = df2.iloc[0] #Delete header in first row df2 = df2.drop('Sample Name', axis=0)  And now it looks like this:  So then I just want to reset the index and name the new column SampleID. df2 = df2.reset_index(names=['SampleID']) But instead of moving the index to a column and naming the column SampleID, it gives the following error:  --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[4], line 21 18 df2 = df2.drop('Sample Name', axis=0) 20 #Reset the indexes ---> 21 df2 = df2.reset_index(names='SampleID') 22 #df2['SampleID'] = df2['SampleID'].str.replace('19OCS-', '') 23 df2['SampleID'] = df2['SampleID'].str.split('.').str[0] File ~\Anaconda3\lib\site-packages\pandas\util\_decorators.py:311, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs) 305 if len(args) > num_allow_args: 306 warnings.warn( 307 msg.format(arguments=arguments), 308 FutureWarning, 309 stacklevel=stacklevel, 310 ) --> 311 return func(*args, **kwargs) TypeError: reset_index() got an unexpected keyword argument 'names'  I originally wrote this code about 6 months ago and it worked just fine but now it's not working? I only changed the file path in the code now vs 6 months ago. I can run the line like this: df2 = df2.reset_index() And it will move the index correctly but, of course, the column is not named.  I can use .rename to change the name: df2 = df2.rename({'index':'SampleID'}, axis=1)  But I can't figure out why the names keyword is not working. I've double checked my syntax and looked at the API and I just cannot figure out why it's giving me this error. I've tried adding/removing brackets; I've tried specifying the level; I've tried using col_fill. It's probably something simple I did wrong. Thank you for your time. Note: It changes from df to df2 after the transpose but I continue working in df2 for the remainder of the code.

Answer: Check that version of pandas you are running is >= 1.5.0. The parameter was introduced in 1.5.0 for reset_index() import pandas as pd pd.__version__
Question: I have a pretty big pandas dataframe and I want to select some rows based on conditions. The problem is that the act of saving as CSV is separate from the overall flow of the program and consumes quite a bit of time. Is it possible to separate the threads so that the main thread progresses to the selected rows, while at the same time unselected rows are saved as csv in another thread? such as... # This is pseudo code import pandas as pd df = pd.DataFrame({"col1":[x for x in range(10000)], "col2":[x**2 for x in range(0, 10000)]}) df_selected = df[df.apply(lambda x: x.col1%3==0, axis=1)] df_unselected = df[df.apply(lambda x: x.col1%3!=0, axis=1)] def Other_thread_save_to_csv(df:pd.DataFrame): # this function is the last function to use df_unselected . Other_thread_save_to_csv(df_unselected ) all_other_hadlings(df_selected ) 

Answer: Yes, Python's either threading or multiprocessing features are handy for concurrent tasks like saving a DataFrame to CSV while doing other tasks. There are things you need to consider while working with threads and multiprocessing in python:  Global Interpreter Lock (GIL) in Python: This means threading may not always speed up CPU-heavy tasks. But for I/O tasks (like file writing), it's quite good to use.  Use Multiprocessing for Heavy CPU Tasks: If your other DataFrame tasks are CPU-intensive, multiprocessing is a better choice than threading.   and the last would be the Thread Safety, You have to Make sure no other thread is altering the DataFrame when you're writing it to a CSV. # This is pseudo code import pandas as pd import threading def save_to_csv(df, filename): df.to_csv(filename, index=False) df = pd.DataFrame({"col1": [x for x in range(10000)], "col2": [x**2 for x in range(10000)]}) df_selected = df[df["col1"] % 3 == 0] df_unselected = df[df["col1"] % 3 != 0] # Initiating a thread to save a portion of DataFrame thread = threading.Thread(target=save_to_csv, args=(df_unselected, 'unselected_rows.csv')) thread.start() # Continue other tasks with the main thread # additional_operations(df_selected) # Optionally, wait for the thread to complete thread.join()  save_to_csv function runs on a separate thread, allowing your program to process df_selected while df_unselected gets saved in the background.
Question: I'm watching the MIT OCW 2008 Introduction to Computer Science and Programming, and trying to do the assignments. This is informal so there's no grading involved. There's this problem where you have to have to buy specific numbers of nuggets with packs containing 6, 9, and 20 nuggets. The aim is to learn to how to use recursions in Python. I'm solving for up to 65 nuggets. I've written the code below. It kind of works, but the problem is, I get exactly one solution for each number I input. How do I make sure that I get all the possible combinations for each number of total nuggets? def testtheorem(x): for a in range(0,15): for b in range(0,15): for c in range(0,15): y = 6*a + 9*b + 20*c if y == x and x < 66: print ("For", int(x), "total;") print ("6 piece can be", int(a)) print ("9 piece can be", int(b)) print ("20 piece can be", int(c)) testtheorem(x+1) return 

Answer: I took a quick go at this question, following the comment from gog. The reason that you are only getting one answer for each answer you input is that you are returning prematurely. Each time you find the first answer, you create a new child function and look for an answer in this number. Eventually, you reach 66 which can never have an answer. Then you start heading back, all the way to the top - to your input number. I drew a small example with starting at 64:  To solve your problem, you need to change at which point in the function you are calling the recursive function and when you are returning. Other than that, your code is on the right track.