eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: values = [1,2,3,2,3,1] colors = ['r','g','b'] expected_output = ['r', 'g', 'b', 'g', 'b', 'r'] # how to create this in pandas? df = pd.DataFrame({'values': values}) df['colors'] = expected_output  I want to make a new column in my dataframe where the colors are selected based on values in an existing column. I remember doing this in xarray with a vectorised indexing trick, but I can't remember if the same thing is possible in pandas. It feels like it should be a basic indexing task. The current answers are a nice start, thanks! They take a bit too much advantage of the numerical nature of "values" though. I'd rather something generic that would also work if say values = ['a', 'b', 'c', 'b', 'c', 'a']  I guess the "map" method probably still works.

Answer: Code use numpy indexing import numpy as np df['colors'] = np.array(colors)[df['values'] - 1]  df  values color 0 1 r 1 2 g 2 3 b 3 2 g 4 3 b 5 1 r  If you want to solve this problem using only Pandas, use map function. (with @Onyambu comment) m = dict(enumerate(colors, 1)) df['colors'] = df['values'].map(m)
Question: I would like to use a MultiIndex in Pandas where at every level I have a nested tuple. I know I could in principle unpack the thing but this would be less legible and annoying. In general, the elements of the tuple (a class name and some parameters) have meaning only together, I would like to make it harder to end up with nonsensical pairs, the tuples have different lengths, and I'd like to use MultiIndex.from_product. Everything works fine when creating the DataFrame and accessing values, but when writing I get results I wasn't expecting. In a simple example, the following code: import pandas as pd index=pd.MultiIndex.from_arrays([[("foo","spam"),("foo","spam")],[("bar","egg"),("bar","egg")],[("baz","bacon"),("pam","bacon")]]) this_index = (("foo","spam"),("bar","egg"),("baz","bacon")) df = pd.DataFrame(index=index, columns=["value"]) print(df) print(df.loc[this_index]) df.loc[this_index]=0 # df.loc[this_index,"value"]=0 print(df)  First prints the table I expected (three tuples as index and NaNs in the column value), then prints the correctly retrieved value NaN, but at the last line shows two extra columns named "bar" and "egg" both set to 0:  value bar egg (foo, spam) (bar, egg) (baz, bacon) 0 0.0 0.0 (pam, bacon) NaN NaN NaN  In this case, using the commented line for the assignment gives the expected result. However, in my case, I need "spam", "egg", and "bacon" to be tuples as well. If I change lines 2 and 3 in the code above putting: index=pd.MultiIndex.from_arrays([[("foo",("spam",)),("foo",("spam",))],[("bar",("egg",)),("bar",("egg",))],[("baz",("bacon",)),("pam",("bacon",))]]) this_index = (("foo",("spam",)),("bar",("egg",)),("baz",("bacon",)))  I have again the expected behaviour with the first two prints, the third gives (now somehow expected):  value bar (egg,) (foo, (spam,)) (bar, (egg,)) (baz, (bacon,)) 0 0.0 0.0 (pam, (bacon,)) NaN NaN NaN  But trying the same workaround as above gives: ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (3, 2) + inhomogeneous part.  And I couldn't find any way to adapt the trick. The best workaround I found at the moment is to use str() on the tuples and then parse again the content if needed, but I feel like there should be a better way. The only trace I found here is an unanswered comment to this answer.

Answer: If I understand correctly, your issue is with this assignments: index=pd.MultiIndex.from_arrays([[("foo",("spam",)),("foo",("spam",))],[("bar",("egg",)),("bar",("egg",))],[("baz",("bacon",)),("pam",("bacon",))]]) this_index = (("foo",("spam",)),("bar",("egg",)),("baz",("bacon",))) df = pd.DataFrame(index=index, columns=["value"]) df.loc[this_index, 'value']=0  Which you can solve using a list for the columns or for the index: df.loc[this_index, ['value']] = 0 # or df.loc[[this_index], 'value'] = 0  Output:  value (foo, (spam,)) (bar, (egg,)) (baz, (bacon,)) 0 (pam, (bacon,)) NaN
Question: I have a large Excel sheet with data. Due to several reasons, I have data that is structured in a transposed way:    Histogram bin 0 bin 1 bin 2 bin 3 bin 4 bin 5     test 1 0 1 5 2 1 0   test 2 0 0 1 7 2 0    When I plot this data, I get 5 series with each 2 values in my plot. What I would like is 2 Series with each 5 points in the plot (displaying the histogram). This corresponds to pressing the "Switch Row/Column" button in Excel. Unfortunately it is not possible to manually press this button for each chartsheet as there are a lot of them. How could I achieve this effect within python? Any help would be greatly appreciated. I have made a simplified version of my code. It uses the same table as the one shown above. I realize using MultiIndex is not required in this case, in my full code however, it is required. import pandas as pd from openpyxl import load_workbook from openpyxl.chart import LineChart, Reference excel_file = 'test.xlsx' # create and write data excel_tests_data = [[0, 1, 5, 2, 1, 0], [0, 0, 1, 7, 2, 0]] df = pd.DataFrame(excel_tests_data) df.columns=pd.MultiIndex.from_tuples([('bin '+str(i), ) for i in range(6)]) df.to_excel(excel_file, startrow=0, sheet_name='test') # open workbook workbook = load_workbook(excel_file) worksheet = workbook.worksheets[0] # create chartsheet chartsheet = workbook.create_chartsheet() chartsheet.title = "Testsheet" chart = LineChart() data = Reference(worksheet, min_col = 2, min_row = 3, max_row = 4, max_col = 7) chart.title = "test" chart.add_data(data) chartsheet.add_chart(chart) # save and close workbook workbook.save(excel_file) workbook.close() 

Answer: If you refer to the Switch Row/Column button, you can simply turn-on the from_rows parameter. chart.add_data(data) chart.add_data(data, from_rows=True) Output (text.xlsx):
Question: I have a 3D numpy array which is really an array of matrices. I want to set the diagonals to zero by using the following approach. When I print the tuple and din it is exactly the same, but it returns different views of the array. m = np.random.normal(0, 0.2, (10, 4, 4)) din = np.diag_indices(m.shape[1], ndim = 2) m[:, np.array([0,1,2,3]), np.array([0,1,2,3])]) # It returns an array of diagonals as expected m[:, tuple(din)] # It returns the array  What did I miss here?

Answer: As described in comments, you need to unpack the indices. Since python 3.11, you can use: m[:, *din]  Output: array([[ 8.61622699e-02, -1.46919069e-01, -9.37771599e-02, 1.94698315e-03], [ 1.60933774e-01, -2.77077615e-02, -1.74135776e-01, -1.72223723e-01], [-1.54804225e-01, 1.08146714e-01, 2.51844877e-01, -2.91622737e-02], [ 1.22213756e-02, 1.59703456e-02, -1.41757563e-01, -5.02470362e-02], [ 1.49296012e-01, -9.60208199e-03, -4.82484338e-01, 1.58012139e-02], [-3.09847219e-01, -1.13959996e-01, -6.71019475e-01, 3.17810448e-01], [ 2.04860543e-04, -2.16311908e-01, 1.39098046e-01, -1.40102017e-01], [-5.82402679e-02, 2.55831587e-01, -3.74597159e-01, 1.23205316e-01], [-1.23942861e-01, 1.40365188e-02, -2.16884333e-02, -2.08800511e-02], [ 1.02934324e-01, -1.81953630e-01, 2.35600757e-01, -2.29315601e-01]])  This syntax is however not supported in older python versions, in this case you can build a single tuple: m[tuple((slice(None), *din))] # or m[(slice(None), *din)]
Question: When I run groupby on a dataframe with as_index set to False, count seems to not work at all. For example, import pandas as pd word_list = ['a', 'b', 'c', 'a', 'c', 'c', 'b', 'a', 'c'] df = pd.DataFrame(word_list, columns=['word']) as_counts = df.groupby('word', as_index=False).count() print(as_counts)   word 0 a 1 b 2 c  If you try the same with .size() as in as_counts = df.groupby('word', as_index=False).size()  it works as expected  word size 0 a 3 1 b 2 2 c 4  What difference between the two functions explains this difference in output? This post is asking a different question entirely as, in this case, the frame is neither MultiIndex nor seeking to maintain non-unique indices. Equally, none of the answers in that post address the problem of this question.

Answer: groupby.count is designed to work on the columns other than the grouping columns while groupby.size is designed to output a single new column (named size), irrespective of the number of existing columns. This makes sense since count depends on the NA status, while size is always constant within a group (the number of rows, irrespective of the values). This can easily be seen by adding several dummy columns: df.assign(dummy1=1, dummy2=None).groupby('word', as_index=False).count() # as many output columns as the input columns (minus grouper) # only the non NA values are counted word dummy1 dummy2 0 a 3 0 1 b 2 0 2 c 4 0  VS: df.assign(dummy1=1, dummy2=None).groupby('word', as_index=False).size() # grouper columns + ONE "size" column word size 0 a 3 1 b 2 2 c 4  This is actually explicitly stated in the documentation:  DataFrameGroupBy.size() Returns: DataFrame or Series Number of rows in each group as a Series if as_index is True or a DataFrame if as_index is False.  So back to your original example, groupby.counts outputs nothing since there are no columns to use (the grouping columns are excluded from the aggregation unless specifically sliced from the DataFrameGroupBy object).  workaround As noted elsewhere, the workaround if you want to count one of the grouping columns is to explicitly slice it: df.groupby('word', as_index=False)['word'].count() word 0 3 1 2 2 4  Amusingly, you can slice it more than once and get as many columns in the output: df.groupby('word', as_index=False)[['word', 'word']].count() word word 0 3 3 1 2 2 2 4 4
Question: My ultimate goal is to write code that only triggers other servers when calling an endpoint and waits for data from a specific channel in Redis to come in. I don't want to know external server's business logic is done due to latency. The call_external_server background task function below will notify the external server to send data to Redis (pub/sub). However, it doesn't execute. Here is my code: async def call_external_server(channel, text): print("call_external_server start") async with aiohttp.ClientSession() as session: async with session.get(f"http://localhost:9000/pub?channel={channel}&text={text}") as resp: print(resp) print("call_external_server finished") return {"response": "external_server is done"} @app.websocket("/ws") async def websocket_endpoint(channel: str, websocket: WebSocket, background_task: BackgroundTasks): await websocket.accept() client_info = dict(websocket.headers) text = client_info.get("text") redis_reader: redis.client.PubSub = await get_redis_pubsub() await redis_reader.subscribe(channel) # Problem is Here background_task.add_task(call_external_server, channel, text) # Background task Doesn't work properly try: while True: message = await redis_reader.get_message(ignore_subscribe_messages=True) if message is not None: decoded_msg = message["data"].decode() if decoded_msg == STOPWORD: print("(Reader) STOP") break await websocket.send_text(decoded_msg) except Exception as e: print(e) await websocket.close() return await websocket.close() return 

Answer: A BackgroundTask is executed after returning a response to a client's request (not on ongoing websocket connections)—see this answer, as well as this answer and this related comment for more details on Background Tasks. As explained by @tiangolo (the creator of FastAPI):  Background Tasks internally depend on a Request, and they are executed after returning the request to the client. In the case of a WebSocket, as the WebSocket is not really "finished" but is an ongoing connection, it wouldn't be a background task.   You could, however, have functions executed in the background, using one of the options described in this answer. If your background task is an async def function, you should rather use Option 3 (see the linked answer above for more details), i.e., using asyncio.create_task(). Have a look at this related answer as well. Example async def call_external_server(channel, text): pass @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): await websocket.accept() # ... asyncio.create_task(call_external_server(channel, text))  I would also suggest having a look at this answer, as well as this answer and this answer on how to spawn an HTTP Client once at application startup and reuse it every time is needed, instead of creating a new connection every time the endpoint is called (as shown in the example provided in your question).
Question: Fresh installed neovim on a new LinuxMint machine with lsp-config, mason plugins and pyright LSP Server (through Mason) and found out that it was not working with lsp.log registering this !!!  .../vim/lsp/rpc.lua:734 "rpc" "/home/manjunath/.local/share/nvim-kick/mason/bin/pyright-langserver" "stderr" "internal/modules/cjs/loader.js:818 throw err; ^ Error: Cannot find module 'node:util' Require stack: - /home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/pyright-langserver.js - /home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/langserver.index.js at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15) at Function.Module._load (internal/modules/cjs/loader.js:667:27) at Module.require (internal/modules/cjs/loader.js:887:19) at require (internal/modules/cjs/helpers.js:74:18) at Object.9632 (/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/pyright-langserver.js:1:557) at o (/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/pyright-langserver.js:1:1142) at Object.1264 (/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/vendor.js:2:794958) at o (/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/pyright-langserver.js:1:1142) at Object.1476 (/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/pyright-internal.js:1:441151) at o (/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/pyright-langserver.js:1:1142) { code: 'MODULE_NOT_FOUND', requireStack: [ '/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/dist/pyright-langserver.js', '/home/manjunath/.local/share/nvim-kick/mason/packages/pyright/node_modules/pyright/langserver.index.js' ] }  Actions taken:  Fresh installed node and npm again. Checked with a sample node app using utils module - WORKING  Outcome:  Working pyright LSP server with Neovim 

Answer: Found out what the issue was. node.js on apt was outdated. Installing latest version from here fixed it.
Question: I am going to do a math competition which allows writing programs to solve problems. The code attempts to solve this problem: WHITE+WATER=PICNIC where each distinct letter represents a different digit. Find the number represented by PICNIC. No imports are allowed (tqdm was just a progress bar). What I tried is below. At the rate of which my computer goes, it is not fast enough for the times competition because it needs to range over 10 to the power of digits there is. Is there a clear solution to this problem? from tqdm import tqdm def find_solution(): for W in tqdm(range(10)): for H in tqdm(range(10), desc='H'): for I in tqdm(range(10), desc='I'): for T in tqdm(range(10)): for E in tqdm(range(10)): for A in (range(10)): for R in (range(10)): for P in (range(1, 10)): # P cannot be 0 for C in (range(10)): for N in (range(10)): white = W * 10000 + H * 1000 + I * 100 + T * 10 + E water = W * 10000 + A * 1000 + T * 100 + E * 10 + R picnic = P * 100000 + I * 10000 + C * 1000 + N * 100 + I * 10 + C if white + water == picnic: return {'W': W, 'H': H, 'I': I, 'T': T, 'E': E, 'A': A, 'R': R, 'P': P, 'C': C, 'N': N} return None solution = find_solution() if solution: print("Solution found:") print(solution) else: print("No solution found.") 

Answer: Observations:  P=1 due to a carry to its sixth digit W must be 5-9 due to a carry being generated (1) No letter can be the same value as another letter.  def find_solution(): P = 1 for W in range(5,10): for H in range(10): if H in (P,W): continue for I in range(10): if I in (H,P,W): continue for T in range(10): if T in (I,H,P,W): continue for E in range(10): if E in (T,I,H,P,W): continue for A in range(10): if A in (E,T,I,H,P,W): continue for R in range(10): if R in (A,E,T,I,H,P,W): continue for C in range(10): if C in (R,A,E,T,I,H,P,W): continue for N in range(10): if N in (C,R,A,E,T,I,H,P,W): continue white = W * 10000 + H * 1000 + I * 100 + T * 10 + E water = W * 10000 + A * 1000 + T * 100 + E * 10 + R picnic = P * 100000 + I * 10000 + C * 1000 + N * 100 + I * 10 + C if white + water == picnic: print(f' WHITE= {white}') print(f' WATER= {water}') print(f'PICNIC={picnic}') print() find_solution()  Output (2 solutions, ~1 second):  WHITE= 83642 WATER= 85427 PICNIC=169069 WHITE= 85642 WATER= 83427 PICNIC=169069  Faster, of course, if you can use libraries (~0.3 sec). Note the documentation of itertools.permutations has a rough Python-only implementation, but it is slower than the hard-coded version above: from itertools import permutations def find_solution(): P=1 for W,H,I,T,E,A,R,C,N in permutations([0,2,3,4,5,6,7,8,9]): if W < 5: continue white = W * 10000 + H * 1000 + I * 100 + T * 10 + E water = W * 10000 + A * 1000 + T * 100 + E * 10 + R picnic = P * 100000 + I * 10000 + C * 1000 + N * 100 + I * 10 + C if white + water == picnic: print(f' WHITE= {white}') print(f' WATER= {water}') print(f'PICNIC={picnic}') print() find_solution()
Question: Given the following upper triangular matrix, I need to shift the values to the bottom as shown: input_array = np.array([ [np.nan, 1, 2, 3], [np.nan, np.nan, 4, 5], [np.nan, np.nan, np.nan, 6], [np.nan, np.nan, np.nan, np.nan], ]) result = np.array([ [np.nan, np.nan, np.nan, np.nan], [np.nan, np.nan, np.nan, 3], [np.nan, np.nan, 2, 5], [np.nan, 1, 4, 6], ])  What is the best way to do this in numpy in terms of fewer lines and speed? Here is the solution I've got using meshgrid: import numpy as np n = input_array.shape[0] j, i = np.meshgrid(np.arange(n), np.arange(n)) i2 = i - np.arange(1,n+1)[::-1] result = input_array[i2, j] 

Answer: Try: ind = (~np.isnan(input_array)).argsort(axis=0) print(np.take_along_axis(input_array, ind, axis=0))  Prints: [[nan nan nan nan] [nan nan nan 3.] [nan nan 2. 5.] [nan 1. 4. 6.]]
Question: I have a dataframe looking like that :  2023-12-31 2024-01-31 myvalue1 myvalue2 myvalue1 myvalue2 a 2 6 2 6 b 3 7 3 7 c 4 8 4 8  which can be created by : df = pd.DataFrame(index=['a', 'b', 'c'], data= {'myvalue1':[2,3, 4], 'myvalue2':[6,7,8]})\ .reindex(['myvalue1', 'myvalue2'] * 2, axis=1) \ .set_axis(pd.MultiIndex.from_product([['2023-12-31','2024-01-31'], ['myvalue1', 'myvalue2']]), axis=1)  How can I get the sum in each column in the following format :  myvalue1 myvalue2 2023-12-31 9 21 2024-01-31 9 21  Note that this is only the example where pair of columns are the same. Thank you

Answer: sum the rows and reshape with unstack: out = df.sum().unstack()  Output:  myvalue1 myvalue2 2023-12-31 9 21 2024-01-31 9 21
Question: I have an enum in Python, which I want to make sure that its values are unique. I see that there are 2 ways I can use to achieve it:  Wrapping the class with @verify(UNIQUE) Wrapping the class with @unique  What is the difference with using each one of them? Which one should I use to gain the best performance?

Answer: You can use which ever you prefer. In terms of implementation the both are the same and I mean the same, its literally copy pasted code in enum.py in in the cpython repo. This code is for the @verify(Unique) and this one for the @unique. I would suggest to use @verify if you have other checks as well but if you only want to check for unique it would be better to use @unique since it will only reference the code needed.
Question: I'm new to Python and stackoverflow. So please forgive any shortcomings in my posting. I want to calculate the row average of a specific row(e.g. Bahrain) and I'm having problem achieving it. I used df.loc but It's returning a key error with the name of the country as it doesn't recognize the name. My dataframe look like something like this:  Country 1990 1995 2000 2005 2010 2015 0 Bahrain 5 4 3 2 1 5 1 Maldives 10 9 8 7 6 5 2 Germany 7 4 3 2 1 7 . .... .. .. .. .. .. .. . .... .. .. .. .. .. .. . .... .. .. .. .. .. ..  I wrote this and I get an error.I got a key error with the name of row. mean=df.loc['Bahrain'].mean(axis=1)  I am getting the following error: The error I prefer to have an output in format of numpy.float64.

Answer: You need to specify the Bahrain at the df["Country"]. And also, you cannot mix text with numbers and take average. Here is how you can do it: rowAverage = df[df['Country'] == 'Bahrain'].iloc[:, 1:].mean()
Question: I want to create a column that is a cumulative sum over a group column but the cumulative sum only happens when the 'days' column meets a certain condition. I have come up with what I regard as a "duct tape" solution, there must be a more elegant way. import polars as pl # Create a DataFrame with literal values df = pl.DataFrame({ "days": [0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 6, 7, 1], "amount": [100, 200, 150, 300, 250, 180, 220, 280, 210, 320,21,456,111], "group": ["A", "B", "A", "C", "B", "A", "C", "B", "C", "A","C","B","B"] }) # Display the DataFrame print(df) #My duct tape solution df = ( df .with_columns( pl.when(pl.col("days") > 2) .then(pl.col("amount")) .otherwise(0).alias("3+days_amount") ) .with_columns( pl.col("3+days_amount").cum_sum().over("group").alias("group_cumsum") ) ) print(df) shape: (13, 3) ┌──────┬────────┬───────┐ │ days ┆ amount ┆ group │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str │ ╞══════╪════════╪═══════╡ │ 0 ┆ 100 ┆ A │ │ 1 ┆ 200 ┆ B │ │ 2 ┆ 150 ┆ A │ │ 3 ┆ 300 ┆ C │ │ 4 ┆ 250 ┆ B │ │ … ┆ … ┆ … │ │ 3 ┆ 210 ┆ C │ │ 4 ┆ 320 ┆ A │ │ 6 ┆ 21 ┆ C │ │ 7 ┆ 456 ┆ B │ │ 1 ┆ 111 ┆ B │ └──────┴────────┴───────┘ shape: (13, 5) ┌──────┬────────┬───────┬───────────────┬──────────────┐ │ days ┆ amount ┆ group ┆ 3+days_amount ┆ group_cumsum │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 │ ╞══════╪════════╪═══════╪═══════════════╪══════════════╡ │ 0 ┆ 100 ┆ A ┆ 0 ┆ 0 │ │ 1 ┆ 200 ┆ B ┆ 0 ┆ 0 │ │ 2 ┆ 150 ┆ A ┆ 0 ┆ 0 │ │ 3 ┆ 300 ┆ C ┆ 300 ┆ 300 │ │ 4 ┆ 250 ┆ B ┆ 250 ┆ 250 │ │ … ┆ … ┆ … ┆ … ┆ … │ │ 3 ┆ 210 ┆ C ┆ 210 ┆ 510 │ │ 4 ┆ 320 ┆ A ┆ 320 ┆ 320 │ │ 6 ┆ 21 ┆ C ┆ 21 ┆ 531 │ │ 7 ┆ 456 ┆ B ┆ 456 ┆ 706 │ │ 1 ┆ 111 ┆ B ┆ 0 ┆ 706 │ └──────┴────────┴───────┴───────────────┴──────────────┘  polars expressions seem very elegant generally, hoping there's something I am missing.

Answer: As mentioned by @jqurious, a single pl.DataFrame.with_columns call can be used to compute the conditional cumulative sum. Moreover, the conditional amount can be computed without a pl.when().then().otherwise() construct, by multiplying the amount with the condition. ( df .with_columns( (pl.col("amount") * (pl.col("days") > 2)).cum_sum().over("group").alias("group_sum") ) )