eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a pandas DataFrame that is typically grouped on a level (or several) of the MultiIndex, then summed. This produces a DataFrame where the rows correspond to the unique values in the grouping level(s), as expected. However, for my application, I am trying to allow for there to be no grouping while maintaining the same output format (with just one row in the DataFrame), but sum() behaves differently with and without grouping. Specifically, if I call sum() on the DataFrame without grouping, it returns a Series, which conflates the data types. Example If I have >>> import pandas as pd >>> df = pd.DataFrame( { 'x': [1, 2, 3, 4], 'y': [0.0, 1.1, 5.2, 4.3] }, index = pd.MultiIndex.from_tuples( [('A', 1), ('A', 2), ('B', 1), ('B', 2)], names = ('ind1', 'ind2') ) ) >>> df x y ind1 ind2 A 1 1 0.0 2 2 1.1 B 1 3 5.2 2 4 4.3  Then grouping on ind1 gives me the expected >>> df.groupby(level='ind1')[['x', 'y']].sum() x y ind1 A 3 1.1 B 7 9.5  However, without the grouping, I get a Series >>> df[['x', 'y']].sum() x 10.0 y 10.6 dtype: float64  The problems are that 1). I need to continue processing the output as a DataFrame, and 2). the sum of 'x' is now a float rather than an integer. I am able to produce the desired results using a dummy grouping variable, >>> df['dummy'] = 1 >>> df.groupby('dummy')[['x', 'y']].sum().reset_index(drop=True) x y 0 10 10.6  But I would like to know if there is a cleaner way that is easier to understand / explain.

Answer: sum returns a Series, which is why you have a single dtype. You can use agg passing a list to force a DataFrame output: df.agg(['sum'])  Output:  x y sum 10 10.6  A simpler version of the dummy grouper would be: df.groupby([0]*len(df)).sum()  Output:  x y 0 10 10.6
Question: I added the exception handling as given here (https://github.com/tiangolo/fastapi/discussions/6678) to my code but I want to print the complete request body to see the complete content. However, when I await the request.json() it never terminates. request.json() returns a coroutine, so I need to wait for the coroutine to complete before printing the result. How can I print the content of the request in case an invalid request was sent to the endpoint? Code example from github with 2 changes by me in the error handler and a simple endpoint. import logging from fastapi import FastAPI, Request, status from fastapi.exceptions import RequestValidationError from fastapi.responses import JSONResponse from pydantic import BaseModel app = FastAPI() @app.exception_handler(RequestValidationError) async def validation_exception_handler( request: Request, exc: RequestValidationError ) -> JSONResponse: exc_str = f"{exc}".replace("\n", " ").replace(" ", " ") logging.error(f"{request}: {exc_str}") body = await request.json() # This line was added by me and never completes logging.error(body) # This line was added by me content = {"status_code": 10422, "message": exc_str, "data": None} return JSONResponse( content=content, status_code=status.HTTP_422_UNPROCESSABLE_ENTITY ) class User(BaseModel): name: str @app.post("/") async def test(body: User) -> User: return body 

Answer: When awaiting request.json, you are trying to read the request body (not the response body, as you may have assumed) out from stream; however, this operation has already taken place in your API endpoint (behind the scenes, in your case). Hence, calling await request.json() is allowed only once in the API's request life-cycle, and attempting reading it again will keep on waiting for it indefinitely (since reading the request body got exhausted), which causes the hanging issue. You may find this answer helpful as well, regarding reading/logging the request body (and/or response body) in a FastAPI/Starlette middleware, before passing the request to the endpoint. As for getting the error body from the response in the RequestValidationError exception handler, you would not need to read the request body, but the response body and you can do that by calling exc.body, instead of request.json(), as demonstrated here and here. Please refer to those answers for more details. Example from fastapi.exceptions import RequestValidationError @app.exception_handler(RequestValidationError) async def validation_exception_handler(request: Request, exc: RequestValidationError): return JSONResponse( status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, content=jsonable_encoder({"detail": exc.errors(), # optionally include the errors "body": exc.body, "custom msg": {"Your error message"}}), )
Question: When I designed microservice and made deployment to K8s, I saw that I had problem to get higher utilization for my microservices (max. utilization was only 0.1-0.3 CPU). Do you have best practices, how can we increase microservice CPU utilization? Let me describe the LAB environment:  K8s with 5 nodes  each node with 14 CPU and 128 GB RAM (nodes are build on virtual machines with VMWare) K8s with nginx with setting full log, etc.   Microservice  In python language (GIL limitation for processing in one process, it means max. 1 CPU utilization) I used three pods Interface REST request/response (without addition I/O operation) The processing time per one call is ~100ms    We made performance tests, and you can see these outputs:  Microservice utilization max. 0.1-0.3 CPU in each pod  I expect the issue is, that K8s management (routing, log, …) generate higher utilization of sources and cannot provide high throughput for utilization of our microservices. I think, the best practices for higher utilization of microservices can be: 1] Increase amount of pods  Pros, we will get higher microservice utilization but amount of pods are limited per one K8s node Cons, the utilization of microservice per pod will be still the same  2] Use micro batch processing  Pros, we can support bundling of calls (per e.g. one, two seconds) and in this case, that processing time on microservice side will be higher Cons, we will increase processing time because bundling (not ideal scenario for real-time processing)  3] K8s change log level  Pros, we can decrease level of logs in nginx, … to error Cons, possible issue for detail issue tracking  4] Use K8s nodes with physical HW (not VMware)  Pros, better performance Cons, this change can generate addition costs (new HW) and maintenance  Do you use other best practices, ideas for high microservice utilization in k8s (my aim is to get 0.8-1 CPU per pod for this python code)?

Answer: Performance testing is a very complex topic, it requires a lot of precision when building the testing setup, and solid knowledge for all the building parts, since it's very easy to mess things up (I did that many times). Couple of ideas from my side:  If you run a single-threaded app on a pod with more than 1 CPU configured, then you'll never see high CPU usage on a pod level. Even if you run a multi-threaded app that has a heavy I/O-bound workload (lots of external HTTP calls for example), you'll still not see high CPU usage, since the threads will be most of the time in a non-runnable state. Kubernetes management workflows do have some overhead that can be observed when looking on cluster-level (or even node-level) metrics but pod-level metrics are fully related to your application (especially CPU usage).  So to see high CPU usage on a pod-level, you can do 2 things:  Run a single-threaded app (that does CPU-heavy tasks) with a pod configured with 1 CPU If you have a multi-threaded app, the pod CPU cores should be the same with the number of threads in your app (and the app workload should be CPU-bound, of course), to get the max CPU usage.
Question: I have a data frame with the following nesting in Pyspark: >content: struct importantId: string >data: array >element: struct importantCol0: string importantCol1: string  I need the following output:     importantId importantCol0 importantCol1     10800005 0397AZ 0397AZ   10800006 0397BZ 0397BZ     I tried the following code:  df1 = df0.select(F.col('content.*')) I got:     importantId data     10800005 {importantCol0: 0397AZ, importantCol1: 0397AZ}   10800006 {importantCol0: 0397BZ, importantCol1: 0397BZ}     I followed with: df2 = df1.select(F.col('importantId'), F.col('data.*') but I get the following error: AnalysisException: Can only star expand struct data types. Attribute: ArrayBuffer(data). Does anyone know how to fix this? I was expecting a way to expand an array like a struct

Answer: Let us use inline to explode array of structs to columns and rows result = df.select('*', F.inline('data')).drop('data')  Example, df.show() +------------+--------------------+ |importantCol| data| +------------+--------------------+ | 1| [{1, 2}, {4, 3}]| | 2|[{10, 20}, {40, 30}]| +------------+--------------------+ result.show() +------------+-------------+-------------+ |importantCol|importantCol0|importantCol1| +------------+-------------+-------------+ | 1| 1| 2| | 1| 4| 3| | 2| 10| 20| | 2| 40| 30| +------------+-------------+-------------+
Question: Every example I have seen of Python string variable interpolation uses the print function. For example: num = 6 # str.format method print("number is {}".format(num)) # % placeholders print("number is %s"%(num)) # named .format method print("number is {num}".format(num=num))  Can you interpolate variables into strings without using print?

Answer: Ok...so, it's kind of easy. The old method: num = 6 mystr = 'number is %s' % num print(mystr) # number is 6  The newer .format method: num = 6 mystr = "number is {}".format(num) print(mystr) # number is 6  The .format method using named variable (useful for when sequence can't be depended upon): num = 6 mystr = "number is {num}".format(num=num) print(mystr) # number is 6  The shorter f-string method: (thank you @mayur) num = 6 mystr = f"number is {num}" print(mystr) # number is 6
Question: I have the following dataframe:  Country Crop Harvest Year Area (ha) Afghanistan Maize 2019 94910 Afghanistan Maize 2020 140498 Afghanistan Maize 2021 92144 Afghanistan Winter Wheat 2019 2334000 Afghanistan Winter Wheat 2020 2668000 Afghanistan Winter Wheat 2021 1833357 Argentina Maize 2019 7232761 Argentina Maize 2020 7730506 Argentina Maize 2021 8146596 Argentina Winter Wheat 2019 6050953 Argentina Winter Wheat 2020 6729838 Argentina Winter Wheat 2021 6394102 China Maize 2019 41309740 China Maize 2020 41292000 China Maize 2021 43355859 China Winter Wheat 2019 23732560 China Winter Wheat 2020 23383000 China Winter Wheat 2021 23571400 Ethiopia Maize 2019 2274306 Ethiopia Maize 2020 2363507 Ethiopia Maize 2021 2530000 Ethiopia Winter Wheat 2019 1789372 Ethiopia Winter Wheat 2020 1829051 Ethiopia Winter Wheat 2021 1950000 France Maize 2019 1506100 France Maize 2020 1691130 France Maize 2021 1549520 France Winter Wheat 2019 5244250 France Winter Wheat 2020 4512420 France Winter Wheat 2021 5276730 India Maize 2019 9027130 India Maize 2020 9569060 India Maize 2021 9860000 India Winter Wheat 2019 29318780 India Winter Wheat 2020 31357020 India Winter Wheat 2021 31610000 Namibia Maize 2019 21123 Namibia Maize 2020 35000 Namibia Maize 2021 46070 Namibia Winter Wheat 2019 1079 Namibia Winter Wheat 2020 2000 Namibia Winter Wheat 2021 3026  I want to select the top 2 countries by the average value of Area (ha) column across the `Harvest Year's. I tried this but it does not work: df = df.groupby("Crop", dropna=False).apply( lambda x: x.nlargest(2, "Area (ha)") ) Output should be, here China and india are the countries with the largest average Area (ha) for both maize and Winter Wheat, but in the full datasets different countries would have largest values for different crops: Country Crop Harvest Year Area (ha) China Maize 2019 41309740 China Maize 2020 41292000 China Maize 2021 43355859 China Winter Wheat 2019 23732560 China Winter Wheat 2020 23383000 China Winter Wheat 2021 23571400 India Maize 2019 9027130 India Maize 2020 9569060 India Maize 2021 9860000 India Winter Wheat 2019 29318780 India Winter Wheat 2020 31357020 India Winter Wheat 2021 31610000 

Answer: IIUC, you can do double .groupby: x = ( df.groupby("Crop") .apply(lambda x: x.groupby("Country")["Area (ha)"].mean()) .stack() .groupby(level=0, group_keys=False) .nlargest(2) ) print(x)  Prints top 2 Crop/Countries by average area: Crop Country Maize China 4.198587e+07 India 9.485397e+06 Winter Wheat India 3.076193e+07 China 2.356232e+07 dtype: float64   Then you can use this index to filter the original dataframe: out = df.set_index(["Crop", "Country"]).loc[x.index].reset_index() print(out)  Prints:  Crop Country Harvest Year Area (ha) 0 Maize China 2019 41309740 1 Maize China 2020 41292000 2 Maize China 2021 43355859 3 Maize India 2019 9027130 4 Maize India 2020 9569060 5 Maize India 2021 9860000 6 Winter Wheat India 2019 29318780 7 Winter Wheat India 2020 31357020 8 Winter Wheat India 2021 31610000 9 Winter Wheat China 2019 23732560 10 Winter Wheat China 2020 23383000 11 Winter Wheat China 2021 23571400
Question: I have a list of strings that is used to define navigation pane in a Django layout template. I want to use the same list for view function names and use a loop to define urlpatterns in urls.py accordingly. Example: menu = ["register", "login", "logout"] urlpatterns = [path("", views.index, name="index"),]  From the above, I want to arrive at urlpatterns = [ path("", views.index, name="index"), path("/register", views.register, name="register"), path("/login", views.login, name="login"), path("/logout", views.logout, name="logout"), ]  I am able to pass the route as str and kwargs as a dict, but struggling with transforming the string into a name of a callable views function. Is this possible? for i in menu: url = f'"/{i}"' #works when passed as parameter to path() rev = {"name": i} #works when passed as parameter to path() v = f"views.{i}" #this does not work v = include(f"views.{i}", namespace="myapp") #this does not work either urlpatterns.append(path(url, v, rev)) 

Answer: Yes, you can use: menu = ['register', 'login', 'logout'] urlpatterns = [ path('', views.index, name='index'), *[path(f'{name}/', getattr(views, name), name=name) for name in menu], ]
Question: I have a dataclass that contains within it another dataclass: @dataclass class A: var_1: str var_2: int @dataclass class B: var_3: float var_4: A  I would like to create a list of all field names for attributes that aren't dataclasses, and if the attribute is a dataclass the to list the attributes of that class, so in this case the output would be ['var_3', 'var_1', 'var_2'] I know it's possible to use dataclasses.fields to get the fields of a simple dataclass, but I can't work out how to recursively do it for nested dataclasses. Ideally I would like to be able to do it by just passing the class type B (in the same way you can pass the type to dataclasses.fields), rather than an instance of B. Is it possible to do this? Thank you!

Answer: Use dataclasses.fields() to iterate over all the fields, making a list of their names. Use dataclasses.is_dataclass() to tell if a field is a nested dataclass. If so, recurse into it instead of adding its name to the list. from dataclasses import fields, is_dataclass def all_fields(c: type) -> list[str]: field_list = [] for f in fields(c): if is_dataclass(f.type): field_list.extend(all_fields(f.type)) else: field_list.append(f.name) return field_list
Question: I have a pandas DataFrame containing daily dates, and within this DataFrame, some dates are missing. I aim to generate a new time series that includes only the last day of each week from that DataFrame. For instance, if there are only Wednesday and Thursday entries for a specific week, the resulting time series should retain only the Thursday data point for that week. For example I tried the following: import pandas as pd import numpy as np # Create a sample time series with date index #SUNDAY, MONDAY, TUESDAY, WED, THURSDAY date_list = ['2023-10-01', '2023-10-02', '2023-10-03', '2023-10-04', '2023-10-05'] # Convert the date list to a pandas datetime index date_rng = pd.to_datetime(date_list) data = np.random.rand(len(date_rng)) time_series = pd.Series(data, index=date_rng) # Resample the time series to weekly frequency and select the last observation for each week weekly_last = time_series.resample('W').last() weekly_last['Day of the Week'] = weekly_last.index.day_name() # Print the result print(weekly_last)  which prints: ['SUNDAY', 'SUNDAY'] whereas it should print ['SUNDAY', 'THURSDAY'] So I don't really know how to achieve what I want? Thank you very much for your help

Answer: You can use the isocalendar() method to get the week of the year, and then group on that: df = pd.DataFrame({'date': date_rng, 'value': data}) df['week'] = df['date'].dt.isocalendar().week grouped = df.groupby('week').last().set_index('date') weekly_last = grouped['value'].copy()  Then you get the expected result: >>> weekly_last.index.day_name() Index(['Sunday', 'Thursday'], dtype='object', name='date')   When you resample using 'W', you are grouping the data to every week, as you want. However, the assigned label on the index will be the corresponding Sunday. You can pick a different day anchor, but this doesn't help you. So you basically need to do the same grouping by keep your original dates as labels. To do this, you can group on the numbered week of the year, take the last observation each week, and use that date for the index. This requires a few extra steps beyond one call to resample/groupby, because. A couple notes:  If you directly group on the week of the year (time_series.groupby(time_series.index.isocalendar().week).last(), you lose the dates and will only see the week of the year in the index. That's why I collect things in a dataframe - then the dates are preserved in their own column, and you can use set_index() to make them the index. You currently can also use the weekofyear attribute to get the week of the year. But I get a deprecation warning when using this, and an instruction to use isocalendar().
Question: I have created two turtles say 'tur1' and 'tur2' in python using turtle module. I have created event listener on the screen. I have created a function 'move_fwd(turte_name)' which moves the turtle forward by 20 steps. I want that if key 'w' is pressed then this method shoud be called by 'tur1' and tur1 should move and if key 'i' is pressed then this function should be called by 'tur2' turtle and it should move. I dont know how to pass argument 'turtle_name' while this method is called by event_listener. from turtle import Turtle, Screen screen = Screen() tur1 = Turtle() tur1.shape("turtle") tur1.shape("turtle") tur1.penup() tur1.setposition(x=-200, y=-100) tur2 = Turtle() tur2.shape("turtle") tur2.penup() tur2.setposition(x=200, y=100) def move_fwd(turtle_name): turtle_name.forward(20) screen.listen() screen.onkey(move_fwd, "w") # how to tell that this is to be called for tur1 screen.onkey(move_fwd, "i") # how to tell that this is to be called for tur2 screen.exitonclick()  I have some idea how to do this in javascript by adding addEventListener ..  document.addEventListener("keypress", function(event) { makeSound(event.key); buttonAnimation(event.key); }); function makeSound(key) { switch (key) { case "w": var tom1 = new Audio("sounds/tom-1.mp3"); tom1.play(); break; case "a": var tom2 = new Audio("sounds/tom-2.mp3"); tom2.play(); break; case "s": var tom3 = new Audio('sounds/tom-3.mp3'); tom3.play(); break;  But no idea about how to do in python. Thanks for your valuable time.

Answer: Use a lambda screen.onkey(lambda: move_fwd(tur1), "w") screen.onkey(lambda: move_fwd(tur2), "i")
Question: I've a set of student and task which I wish to assign. How can I add an objective function so that I get a balance number of task assigned to each student? Subsequently, I will also add some constraints to restrict certain student(s) from receiving certain task hence I would like to balance out the task per student as much as I could. For example the codes below, I have 5 student and 13 tasks, some student would have received 2 task or 3 task which is the most optimal. The objective function I could think of is using the max number task - min number of task but I am unable to form the logic using the or-tool syntax. Thanks in advance! students = [0,1,2,3,4] tasks = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] model = cp_model.CpModel() max_task = math.ceil(len(tasks)/len(students)) x = {} for student in students: for task in tasks: x[(student ,task)] = model.NewBoolVar(f'student_{student}_task_{task}') # add constraint: each task must be assigned to exactly one student for task in tasks: model.Add(sum(x[student ,task] for student in students) == 1) for student in students: model.Add(sum(x[(student,task)] for task in tasks) > 0) model.Add(sum(x[(student,task)] for task in tasks) <= max_task) # add a objective function here solver = cp_model.CpSolver() status = solver.Solve(model) print(status) if status == cp_model.OPTIMAL or status == cp_model.FEASIBLE: for worker, task in x: if solver.Value(x[(student,task)])==1: print(f'Student {student} is assigned to Task {task}') else: print("No solution found.") 

Answer: You can have a look at the balance_group example. It uses this trick to minimize the spread  e = model.NewIntVar(0, 550, "epsilon") # Constrain the sum of values in one group around the average sum per group. for g in all_groups: model.Add( sum(item_in_group[(i, g)] * values[i] for i in all_items) <= average_sum_per_group + e ) model.Add( sum(item_in_group[(i, g)] * values[i] for i in all_items) >= average_sum_per_group - e )
Question: Do you know what is the equivalent for javascript ? This is my code : def sort_df(datas): df_not_sorted = pd.read_json(StringIO(datas)) df_not_sorted["Date"] = df_not_sorted["Date"].apply(lambda dates: [datetime.fromisoformat(date.rstrip('Z')) for date in dates]) df_not_sorted["time"] = pd.to_datetime(df_not_sorted["time"]) df_not_sorted["time"] = df_not_sorted["time"].dt.tz_localize(None) df = df_not_sorted.set_index(["Bus", "time"]).sort_index() return df 

Answer: You can use Danfo.js:  Danfo.js is heavily inspired by the Pandas library and provides a similar interface and API. This means users familiar with the Pandas API can easily use Danfo.js.  https://danfo.jsdata.org/ https://danfo.jsdata.org/getting-started
Question: It seems that Pyright (the Python type checker made by Microsoft) can only be used as a command line tool or from VS Code. But is it possible to call pyright from code (as an API)? For example, mypy supports usage like: import sys from mypy import api result = api.run("your code") 

Answer: like @Grismar said, this might be an xy problem... if not, here is a general solution: import subprocess command = ['pyright', 'path/to/your/file.py'] result = subprocess.run(command, capture_output=True, text=True) output = result.stdout print(output)
Question: I have a dataframe containing costs for time periods:     valid_from valid_to cost     2018-10-09 23:00:00 2019-09-30 23:00:00 28.6700   2019-09-30 23:00:00 2021-03-18 00:00:00 26.2700   2022-10-13 23:00:00 NaT 39.7339     If 'valid_to' is NaT it means the cost is still current. I want to add the correct cost to the time periods in the second dataframe which is broken into 30 minute periods:     valid_from valid_to consumption cost     2023-09-16 23:30:00 2023-09-17 00:00:00 0.040 39.7339   2023-09-17 00:00:00 2023-09-17 00:30:00 0.030 39.7339   2019-10-17 00:30:00 2019-10-17 01:00:00 0.030 26.2700   2018-10-16 20:30:00 2018-10-16 21:00:00 0.030 28.6700     How do I achieve this?

Answer: The main idea is to get cost for rows where valid_from and valid_to overlap between the two dataframes - this is a form of inequality join which is effectively handled by conditional_join: # pip install pyjanitor import pandas as pd import janitor # fill df1's valid_to with a timestamp in the future df1.valid_to = df1.valid_to.fillna(pd.Timestamp.max) (df1 .conditional_join( df2, ('valid_from', 'valid_from', '<='), ('valid_to', 'valid_to', '>='), df_columns='cost') #.move(source='cost',position='after',axis=1) ) cost valid_from valid_to consumption 0 28.6700 2018-10-16 20:30:00 2018-10-16 21:00:00 0.03 1 26.2700 2019-10-17 00:30:00 2019-10-17 01:00:00 0.03 2 39.7339 2023-09-16 23:30:00 2023-09-17 00:00:00 0.04 3 39.7339 2023-09-17 00:00:00 2023-09-17 00:30:00 0.03