eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm developing a little testing website using the OpenAI API. I'm trying to stream GPT's response, just like how it's done on https://chat.openai.com/chat. This works just fine when running my Flask application on a local development server, but when I deploy this app to Google Cloud, the response is given in one go, instead of being streamed. I have tried disabling buffering according to https://cloud.google.com/appengine/docs/flexible/how-requests-are-handled?tab=python#x-accel-buffering, but that didn't resolve the issue. I suspect that my issue lies in how I'm configuring my app on Google Cloud (or the lack thereof). This is what I've got going on currently, this works when running the application locally. main.py @app.route('/stream_response', methods=['POST']) def stream_response(): prompt_text = request.form['prompt'] def generate(): for chunk in gpt_model.get_response(prompt_text, stream=True): for choice in chunk['choices']: dictionary: dict = choice['delta'] if 'content' in dictionary: yield dictionary['content'] response = Response(generate(), content_type='text/html') response.headers['X-Accel-Buffering'] = 'no' return response  prompt.html <script> function streamResponse() { var promptText = document.getElementById("prompt").value; var xhr = new XMLHttpRequest(); xhr.open("POST", "/stream_response", true); xhr.setRequestHeader("Content-Type", "application/x-www-form-urlencoded"); xhr.onprogress = function () { document.getElementById("response-container").style.display = "block"; document.getElementById("response").innerHTML = xhr.responseText; console.log(xhr.responseText) }; xhr.send("prompt=" + encodeURIComponent(promptText)); } </script>  Google Cloud app.yaml runtime: python310 handlers: - url: /.* script: auto  Google Cloud deployment process gcloud app deploy 

Answer: From your app.yaml, it means you're deploying to Google App Engine (GAE) Standard Environment. GAE doesn't support streaming - see doc where it says  App Engine does not support streaming responses where data is sent in incremental chunks to the client while a request is being processed. All data from your code is collected as described above and sent as a single HTTP response.  Chat UIs also usually require sockets. GAE Standard doesn't support web sockets but GAE Flex does (see docs)
Question: I have a dataframe and I need to filter out who is the owner of which books so we can send them notifications. I am having trouble merging the data in the format I need. Existing dataframe     Book Owner     The Alchemist marry   To Kill a Mockingbird john   Lord of the Flies abel   Catcher in the Ry marry   Alabama julia;marry   Invisible Man john     I need to create new dataframe that lists the owners in column A and all the books they own in Column B. Desired output     Owners Books     marry The Alchemist, Catcher in the Ry, Alabama   john To Kill a Mockingbird, Invisible Man   abel Lord of the Flies   julia Alabama     I tried creating 2 dfs from and then merging but the results are never accurate. Anyone know a more efficient way to do this? Current code not working: from pathlib import Path import pandas as pd file1 = Path.cwd() / "./bookgrid.xlsx" df1 = pd.read_excel(file1) df2 = pd.read_excel(file1) ##Perfrom the Vlookup Merge merge = pd.merge(df1, df2, how="left") merge.to_excel("./results.xlsx") 

Answer: You need to split, explode, groupby.agg: (df.assign(Owner=lambda d: d['Owner'].str.split(';')) .explode('Owner') .groupby('Owner', as_index=False, sort=False).agg(', '.join) )  NB. if you need the plural form in the column headers, add .add_suffix('s') or .rename(columns={'Book': 'Books', 'Owner': 'Owners'}). Output:  Owner Book 0 marry The Alchemist, Catcher in the Ry, Alabama 1 john To Kill a Mockingbird, Invisible Man 2 abel Lord of the Flies 3 julia Alabama
Question: I would like to get values from column n where values in subset of other columns is True. Example, the data frame: t, f = True, False data = [ [t, f, f, '1'], [f, f, f, '2'], [f, t, f, '3'], [f, f, t, '4'] ] df = pd.DataFrame(data, columns=list("abcn"))  df as table  a b c n 0 True False False 1 1 False False False 2 2 False True False 3 3 False False True 4  columns for search is a and b, and I wish to get records from n where these columns are True, what I tried: fcols = ("a", "b") df[df[[*fcols]] == t].dropna(axis=0, how='all')  this is give me right records, but with Nan in the column n  a b c n 0 True NaN NaN NaN 2 NaN True NaN NaN  I'm feel that I'm more or less close to the solution, but ...

Answer: Use any to aggregate the booleans for your boolean indexing: fcols = ("a", "b") out = df[df[[*fcols]].eq(t).any(axis=1)]#.dropna(axis=0, how='all') # dropna not needed  Output:  a b c n 0 True False False 1 2 False True False 3  Intermediate indexing Series: df[[*fcols]].eq(t).any(axis=1) 0 True 1 False 2 True 3 False dtype: bool
Question: I have a simple Dockerfile FROM python:3.10-alpine # Set environment variables ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 # Set arguments ARG USERNAME=jpg ARG USER_DIR=/home/$USERNAME ARG WORK_DIR=$USER_DIR/app # Creating a non-root user RUN adduser -S $USERNAME # Switching the user USER $USERNAME # Create a new directory to keep the project files RUN mkdir $WORK_DIR # Copy local contents into container COPY ./ $WORK_DIR # Setting newly created directory as PWD WORKDIR $WORK_DIR # Adding user's bin path to `PATH` variable ENV PATH "$PATH:$USER_DIR/.local/bin" # Installing pip packages RUN pip install pip -U  I built the image using following command docker build . -t test-img:latest  After the successfull image build, run the image using docker run -i -t test-img:latest python  This gave me an interactive shell, as expected. Also, I tried to pull the user info using getuser(...) function as well Python 3.10.10 (main, Mar 14 2023, 03:01:12) [GCC 12.2.1 20220924] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import getpass >>> getpass.getuser() 'jpg' >>>  Then, I moved to another terminal and logged in to the same container as root user. After logged-in, tried to switch the user to jpg (which is defined within the Dockerfile) $ docker exec -it -u root f6dc515bca29 sh /home/jpg/app # su - jpg This account is not available /home/jpg/app # su - "jpg" This account is not available /home/jpg/app # su -l "jpg" This account is not available /home/jpg/app # su -l jpg This account is not available /home/jpg/app #   Question How can I switch the user to jpg (or any non-root user) in an alpine-based container from root? Note: It is mandatory to log in as root (-u root) in an exec session because this is to simulate a situation of AWS Fargate containers.

Answer: You're getting the This account is not available message because running adduser -S explicitly sets the shell for the account to /sbin/nologin (because you've asked for a system account with -S, the assumption is you don't expect interactive logins for this user). You can explicitly set the shell for the named user: / # adduser -S -s /bin/sh jpg / # su - jpg 710ed667ddb2:~$ whoami jpg  ...or, as @tripleee suggests, just drop the -S from your adduser invocation and create a normal user; in this case, the shell will default to /bin/sh.
Question: Working in Databricks, I've got a dataframe which looks like this: columns = ["a", "b", "c"] data = [(True, True, True), (True, True, True), (True, False, True)] df = spark.createDataFrame(data).toDF(*columns) df.display()   I'd like to select only those columns of the dataframe in which not all values are True. In pandas, I would use df['a'].all() to check whether all values of column "a" are True. Unfortunately, I don't find an equivalent in PySpark. I have found a solution for the problem, but it seems much too complicated: df.select(*[column for column in df.columns if df.select(column).distinct().collect() != spark.createDataFrame([True], 'boolean').toDF(column).collect()])  The solution returns what I want:  Is there an easier way of doing this in PySpark?

Answer: Here is one approach which follows a similar operational model to that of pandas def is_all(c): return (F.sum(F.col(c).astype('int')) == F.count(c)).alias(c) # create a boolean mask, # for e.g in pandas this would similar to df.all(axis=0) mask = df.agg(*[is_all(c) for c in columns]).collect()[0] # Use the boolean mask to filter the columns result = df.select(*[c for c in columns if not mask[c]])  Result +-----+ | b| +-----+ | true| | true| |false| +-----+
Question: Update: The vectorization rules have since been formalized. The query runs as expected without warning.  For a dataframe, the goal is to have the mean of a column - a group_by another column - b given the first value of a in the group is not null, if it is, just return null. The sample dataframe df = pl.DataFrame({"a": [None, 1, 2, 3, 4], "b": [1, 1, 2, 2, 2]})  I tried something like df.group_by("b").agg( pl.when(pl.col("a").first().is_null()).then(None).otherwise(pl.mean("a")) )  The results are as expected but get a warning saying when may not be guaranteed to do its job in group_by context. The predicate 'col("a").first().is_null()' in 'when->then->otherwise' is not a valid aggregation and might produce a different number of rows than the groupby operation would. This behavior is experimental and may be subject to change shape: (2, 2) ┌─────┬─────────┐ │ b ┆ literal │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═════╪═════════╡ │ 1 ┆ null │ │ 2 ┆ 3.0 │ └─────┴─────────┘  May I know why and what could be a better alternative way to do if-else in group_by?

Answer: You can use:  pl.col("a").is_null().first()  instead of:  pl.col("a").first().is_null()  If we look at both approaches: df.group_by("b", maintain_order=True).agg( pl.col("a"), pl.col("a").is_not_null().alias("yes"), pl.col("a").first().is_not_null().alias("no"), )  shape: (2, 4) ┌─────┬───────────┬────────────────────┬───────┐ │ b ┆ a ┆ yes ┆ no │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ list[i64] ┆ list[bool] ┆ bool │ ╞═════╪═══════════╪════════════════════╪═══════╡ │ 1 ┆ [null, 1] ┆ [false, true] ┆ false │ │ 2 ┆ [2, 3, 4] ┆ [true, true, true] ┆ true │ └─────┴───────────┴────────────────────┴───────┘  My understanding is, that in the no case, only null and 2 are passed to .is_not_null() - the rest of the inputs have been "silently discarded". polars knows a has lengths of 2 and 3 and expects "boolean masks" of the same length. We can take the .first() value of yes which has the same end result: df.group_by("b", maintain_order=True).agg( pl.col("a"), pl.col("a").is_not_null().first().alias("yes"), pl.col("a").first().is_not_null().alias("no"), )  shape: (2, 4) ┌─────┬───────────┬───────┬───────┐ │ b ┆ a ┆ yes ┆ no │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ list[i64] ┆ bool ┆ bool │ ╞═════╪═══════════╪═══════╪═══════╡ │ 1 ┆ [null, 1] ┆ false ┆ false │ │ 2 ┆ [2, 3, 4] ┆ true ┆ true │ └─────┴───────────┴───────┴───────┘  But now all of the inputs have been passed to .is_not_null() and the length check passes.
Question: Each group only has one valid or not_null value in a random row. How do you fill each group with that value? import polars as pl data = { 'group': ['1', '1', '1', '2', '2', '2', '3', '3', '3'], 'col1': [1, None, None, None, 3, None, None, None, 5], 'col2': ['a', None, None, None, 'b', None, None, None, 'c'], 'col3': [False, None, None, None, True, None, None, None, False] } df = pl.DataFrame(data)  shape: (9, 4) ┌───────┬──────┬──────┬───────┐ │ group ┆ col1 ┆ col2 ┆ col3 │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str ┆ bool │ ╞═══════╪══════╪══════╪═══════╡ │ 1 ┆ 1 ┆ a ┆ false │ │ 1 ┆ null ┆ null ┆ null │ │ 1 ┆ null ┆ null ┆ null │ │ 2 ┆ null ┆ null ┆ null │ │ 2 ┆ 3 ┆ b ┆ true │ │ 2 ┆ null ┆ null ┆ null │ │ 3 ┆ null ┆ null ┆ null │ │ 3 ┆ null ┆ null ┆ null │ │ 3 ┆ 5 ┆ c ┆ false │ └───────┴──────┴──────┴───────┘  Desired output: shape: (9, 4) ┌───────┬──────┬──────┬───────┐ │ group ┆ col1 ┆ col2 ┆ col3 │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str ┆ bool │ ╞═══════╪══════╪══════╪═══════╡ │ 1 ┆ 1 ┆ a ┆ false │ │ 1 ┆ 1 ┆ a ┆ false │ │ 1 ┆ 1 ┆ a ┆ false │ │ 2 ┆ 3 ┆ b ┆ true │ │ 2 ┆ 3 ┆ b ┆ true │ │ 2 ┆ 3 ┆ b ┆ true │ │ 3 ┆ 5 ┆ c ┆ false │ │ 3 ┆ 5 ┆ c ┆ false │ │ 3 ┆ 5 ┆ c ┆ false │ └───────┴──────┴──────┴───────┘  In pandas, I can do the following for each column import pandas as pd df = pd.DataFrame(data) df.col1 = df.groupby('group').col.apply(lambda x: x.ffill().bfill())  How do you do this in polars, ideally with a window function (.over()) ?

Answer: The immediate way to do exactly what you asked is (and it looks the most like your pandas approach): df.with_columns(pl.exclude('group').forward_fill().backward_fill().over('group'))  using pl.all() instead of pl.exclude('group') also works but it'll save some theoretical time by not making it look through the group column for the fills. If there's a list of columns you want to do this to (as opposed to all but group) then you can replace the pl.exclude with a generator or list comprehension cols=['col1','col2','col3'] df.with_columns(pl.col(x).forward_fill().backward_fill().over('group') for x in cols)  You can even use regex in pl.col as long as you use the ^ and $ anchor. df.with_columns(pl.col("^col\d$").forward_fill().backward_fill().over('group'))  Another approach besides forward/backward fills: df.with_columns(pl.col("^col\d$").drop_nulls().first().over('group'))  If the first looks a little weird it's because the drop_nulls is going to return a different number of rows than the original df which will cause an error. If the expression is an aggregation (like sum, min, max, etc) then it doesn't complain about getting a different number of rows and, instead, just propagates that answer to all the rows. In this case first is the aggregation which just means the first thing it sees. Since the filter is only returning one thing we just need a way to tell it to propagate that. The different column selection tricks work in this approach too but I'll spare the reader the extra copy/paste Final note: if your next step is to take unique then you should just do it as a df.group_by to start with df \ .group_by('group', maintain_order=True) \ .agg(pl.col("^col\d$").drop_nulls().first())
Question: I have been developing some codes in pandas, yet I found the executions in pandas are a bit too slow. I then stumbled upon polars, which claims to be blazingly fast and much faster than pandas. I have thus been trying to transfer my existing codes to polars. I am currently working on some stock data, where I have to find out the minimum price of the previous n transaction days. Note that here the transaction dates are discontinuous, since there are Saturdays, Sundays and public holidays. The problem is that polars do not know how to find the previous transaction days, instead polars find the actual previous days. Let's say now we have a DataFrame, and n = 3: df = pl.from_repr(""" ┌────────────┬───────┐ │ Date ┆ Price │ │ --- ┆ --- │ │ date ┆ i64 │ ╞════════════╪═══════╡ │ 2023-01-01 ┆ 1 │ │ 2023-01-02 ┆ 2 │ │ 2023-01-03 ┆ 3 │ │ 2023-01-05 ┆ 4 │ │ 2023-01-10 ┆ 5 │ │ 2023-01-11 ┆ 6 │ │ 2023-01-12 ┆ 7 │ └────────────┴───────┘ """)  The expected output I get from the following codes using pandas: df.to_pandas().Price.rolling(3).min()  0 NaN 1 NaN 2 1.0 3 2.0 4 3.0 5 4.0 6 5.0 Name: Price, dtype: float64  However, when using polars, since there are no indices, the group by function works differently. It sets the window based on the actual physical dates(continuous) instead of the transaction dates(discontinuous, pandas can handle that with indices), which yields the following results: ┌────────────┬───────┐ │ Date ┆ Price │ │ --- ┆ --- │ │ date ┆ i64 │ ╞════════════╪═══════╡ │ 2023-01-01 ┆ 1 │ │ 2023-01-02 ┆ 1 │ │ 2023-01-03 ┆ 1 │ │ 2023-01-05 ┆ 3 │ │ 2023-01-10 ┆ 5 │ │ 2023-01-11 ┆ 5 │ │ 2023-01-12 ┆ 5 │ └────────────┴───────┘  with the following code: df.rolling("Date", period="3d").agg(pl.col("Price").min())  Is there a way to get the desired result, i.e. same as the one in pandas, with the polars library? (Should I manually create an index column???) Or is switching over to polars a bad idea for data like this, since polars advocates for the idea of no indices?

Answer: Polars has dedicated .rolling_* expressions, e.g. .rolling_min() df.with_columns(roll = pl.col("Price").rolling_min(3))  shape: (7, 3) ┌────────────┬───────┬──────┐ │ Date ┆ Price ┆ roll │ │ --- ┆ --- ┆ --- │ │ date ┆ i64 ┆ i64 │ ╞════════════╪═══════╪══════╡ │ 2023-01-01 ┆ 1 ┆ null │ │ 2023-01-02 ┆ 2 ┆ null │ │ 2023-01-03 ┆ 3 ┆ 1 │ │ 2023-01-05 ┆ 4 ┆ 2 │ │ 2023-01-10 ┆ 5 ┆ 3 │ │ 2023-01-11 ┆ 6 ┆ 4 │ │ 2023-01-12 ┆ 7 ┆ 5 │ └────────────┴───────┴──────┘  For the frame level rolling - you can add an index column as you suggested. It's a bit different as we have to use the "format language" to specify the period - instead of a numeric window size. (df.with_row_index() .rolling("index", period="3i") .agg(pl.col("Price").min()) )  shape: (7, 2) ┌───────┬───────┐ │ index ┆ Price │ │ --- ┆ --- │ │ u32 ┆ i64 │ ╞═══════╪═══════╡ │ 0 ┆ 1 │ │ 1 ┆ 1 │ │ 2 ┆ 1 │ │ 3 ┆ 2 │ │ 4 ┆ 3 │ │ 5 ┆ 4 │ │ 6 ┆ 5 │ └───────┴───────┘  You would also need to combine the result with the original frame, generally with a .join()
Question: I have an endpoint which returns a Pydantic object. However, I would like a response code other than 200 in some cases (for example if my service in not healthy). How can I achieve that with FastAPI? class ServiceHealth(BaseModel): http_ok: bool = True database_ok: bool = False def is_everything_ok(self) -> bool: return self.http_ok and self.database_ok @router.get("/health") def health() -> ServiceHealth: return ServiceHealth() 

Answer: You can return a Response Directly. For example, you can use JSONResponse and set the status manually: @router.get("/health") async def health() -> ServiceHealth: response = ServiceHealth() if response.is_everything_ok(): return JSONResponse(content=response.dict(), status_code=200) return JSONResponse(content=response.dict(), status_code=500)  Also, there is a handy status.py which you can import from fastapi that contains all the available status codes: from fastapi import status @router.get("/health") async def health() -> ServiceHealth: response = ServiceHealth() if response.is_everything_ok(): return JSONResponse(content=response.dict(), status_code=status.HTTP_200_OK) return JSONResponse(content=response.dict(), status_code=status.HTTP_500_INTERNAL_SERVER_ERROR)
Question: I'm following the examples in the docs but using Virginia's parcel shp file. Warning: it's about 1GB zipped and 1.8GB unzipped. I have very simply fiava = fiona.open("VirginiaParcel.shp/VirginiaParcel.shp", layer='VirginiaParcel')  from which I can do fiava.schema to get # {'properties': {'FIPS': 'str:8', # 'LOCALITY': 'str:64', # 'PARCELID': 'str:64', # 'PTM_ID': 'str:64', # 'LASTUPDATE': 'date', # 'VGIN_QPID': 'str:50'}, # 'geometry': 'Polygon'}  So far so good but when I do fiava[0] ## I get a Feature object, not the data ## <fiona.model.Feature at 0x7f2fd582aa10>  In the docs it shows this output {'geometry': {'coordinates': [[(-4.663611, 51.158333), (-4.669168, 51.159439), (-4.673334, 51.161385), (-4.674445, 51.165276), (-4.67139, 51.185272), (-4.669445, 51.193054), (-4.665556, 51.195), (-4.65889, 51.195), (-4.656389, 51.192215), (-4.646389, 51.164444), (-4.646945, 51.160828), (-4.651668, 51.159439), (-4.663611, 51.158333)]], 'type': 'Polygon'}, 'id': '1', 'properties': OrderedDict([('CAT', 232.0), ('FIPS_CNTRY', 'UK'), ('CNTRY_NAME', 'United Kingdom'), ('AREA', 244820.0), ('POP_CNTRY', 60270708.0)]), 'type': 'Feature'}  If I use the schema for the specific keys then I can get data one value at a time but this is not optimal fiava[0]['properties']['FIPS'] ## 51149  Even if I do fiava[0].items() then it's just an ItemsView What am I missing?

Answer: You can use fiona.model.to_dict() from fiona.model import to_dict d = to_dict(fiava[0])  if you want to serialize the data as json without calling to_dict() on each object, you can use the fiona.model.ObjectEncoder: import json from fiona.model import ObjectEncoder j = json.dumps(fiava, cls=ObjectEncoder)
Question: I am building an environment in the maintained fork of gym: Gymnasium by Farama. In my gym environment, I state that the action_space = gym.spaces.Discrete(5) and the observation_space = gym.spaces.MultiBinary(25). Running the environment with the agent-environment loop suggested on the Gym Basic Usage website runs with no problems: I registered the environment and it is simply callable by gym.make(). However, I want to now train a reinforcement learning agent on this environment. Now I have come across Stable Baselines3, which makes a DQN agent implementation fairly easy. However, it does seem to support the new Gymnasium. Namely: import gymnasium as gym from stable_baselines3.ppo.policies import MlpPolicy from stable_baselines3 import DQN env = gym.make("myEnv") model = DQN(MlpPolicy, env, verbose=1)  Yes I know, "myEnv" is not reproducable, but the environment itself is too large (along with the structure of the file system), but that is not the point of this question This code produces an error: AssertionError: The algorithm only supports (<class 'gym.spaces.discrete.Discrete',) as action spaces but Discrete(5) was provided  My question is the following: does Stable Baselines3 support Gymnasium? I have tried to instead use gym.spaces in order to define the action_space and observation_space, such that from gym.spaces import Discrete, MultiBinary action_space = Discrete(5) observation_space = MultiBinary(25)  but along with this, I have to rewrite a large portion of the environment to support the old gym package. I wonder whether there is a better solution than that.

Answer: I was a bit confused by the other answer here as I was sure I'd seen gymnasium in the Stable Baselines 3 docs somewhere. Sure enough, it's even in the most basic "getting started" example: https://stable-baselines3.readthedocs.io/en/master/guide/quickstart.html. However, I just tried running exactly this code and received the same error as the OP. Replacing gymnasium with gym 0.21 in the same example works without a problem. Edit: The "getting started" example using gymnasium works with stable_baselines3 version 2.0.0a1 and above.