eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to move from pandas to polars but I am running into the following issue. df = pl.DataFrame( { "integer": [1, 2, 3], "date": [ "2010-01-31T23:00:00+00:00", "2010-02-01T00:00:00+00:00", "2010-02-01T01:00:00+00:00" ] } ) df = df.with_columns( pl.col("date").str.to_datetime().dt.convert_time_zone("Europe/Amsterdam") )  Yields the following dataframe: shape: (3, 2) ┌─────────┬────────────────────────────────┐ │ integer ┆ date │ │ --- ┆ --- │ │ i64 ┆ datetime[μs, Europe/Amsterdam] │ ╞═════════╪════════════════════════════════╡ │ 1 ┆ 2010-02-01 00:00:00 CET │ │ 2 ┆ 2010-02-01 01:00:00 CET │ │ 3 ┆ 2010-02-01 02:00:00 CET │ └─────────┴────────────────────────────────┘  As you can see, I transformed the datetime string from UTC to CET succesfully. However, if I try to cast to pl.Date (as suggested here), it seems to extract the date from the UTC string even though it has been transformed, e.g.: df = df.with_columns( pl.col("date").cast(pl.Date).alias("valueDay") )  shape: (3, 3) ┌─────────┬────────────────────────────────┬────────────┐ │ integer ┆ date ┆ valueDay │ │ --- ┆ --- ┆ --- │ │ i64 ┆ datetime[μs, Europe/Amsterdam] ┆ date │ ╞═════════╪════════════════════════════════╪════════════╡ │ 1 ┆ 2010-02-01 00:00:00 CET ┆ 2010-01-31 │ # <- NOT OK │ 2 ┆ 2010-02-01 01:00:00 CET ┆ 2010-02-01 │ │ 3 ┆ 2010-02-01 02:00:00 CET ┆ 2010-02-01 │ └─────────┴────────────────────────────────┴────────────┘  The valueDay should be 2010-02-01 for all 3 values. Can anyone help me fix this? A pandas dt.date like way to approach this would be nice. By the way, what is the best way to optimize this code? Do I constantly have to assign everything to df or is there a way to chain all of this?

Answer: Update: .dt.date() has since been added to Polars. import polars as pl df = pl.DataFrame( { "integer": [1, 2, 3], "date": [ "2010-01-31T23:00:00+00:00", "2010-02-01T00:00:00+00:00", "2010-02-01T01:00:00+00:00", ], } ) df = df.with_columns( pl.col("date").str.to_datetime().dt.convert_time_zone("Europe/Amsterdam") ) df = df.with_columns( pl.col("date").dt.date().alias("valueDay"), pl.col("date").dt.day().alias("day"), pl.col("date").dt.month().alias("month"), pl.col("date").dt.year().alias("year"), )  shape: (3, 6) ┌─────────┬────────────────────────────────┬────────────┬─────┬───────┬──────┐ │ integer ┆ date ┆ valueDay ┆ day ┆ month ┆ year │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ datetime[μs, Europe/Amsterdam] ┆ date ┆ i8 ┆ i8 ┆ i32 │ ╞═════════╪════════════════════════════════╪════════════╪═════╪═══════╪══════╡ │ 1 ┆ 2010-02-01 00:00:00 CET ┆ 2010-02-01 ┆ 1 ┆ 2 ┆ 2010 │ │ 2 ┆ 2010-02-01 01:00:00 CET ┆ 2010-02-01 ┆ 1 ┆ 2 ┆ 2010 │ │ 3 ┆ 2010-02-01 02:00:00 CET ┆ 2010-02-01 ┆ 1 ┆ 2 ┆ 2010 │ └─────────┴────────────────────────────────┴────────────┴─────┴───────┴──────┘
Question: A is co occurrence dataframe. Why it shown AttributeError: module 'networkx' has no attribute 'from_numpy_matrix' import numpy as np import networkx as nx import matplotlib A=np.matrix(coocc) G=nx.from_numpy_matrix(A) 

Answer: It was updated to nx.from_numpy_array(A)
Question: With pip list, we can see the install packages in our environment. There is no problem with that. We can also write them to a req.txt file with pip freeze and quickly load them in other environments with this req.txt file. My question here is, for example, when we install pandas, libraries such as numpy are installed with them, and we can see other libraries installed outside of pandas with pip list. Can I use an option with pip list to see only the main libraries I have installed here? For example, is there an option to show only pandas (no libraries like numpy) when I do a pip list?

Answer: You can use --not-required flag. This will list packages that are not dependencies of installed packagespip.pypa.io. python -m pip list --not-required  or if pip is in $PATH pip list --not-required
Question: I am using the statsforecast package to fit an AUTOarima model with an external regressor, which works fine. I need to get the model parameters and modify the parameter for the external regressor and rerun the model for scenario analysis. I also need a model summary to provide with my research. How can I get the model specification/parameters and/or a summary from the fitted model in the statsforecast package? A similar questions has been asked on Github (https://github.com/Nixtla/statsforecast/issues/72) but remains unanswerd as of now. I looked through the documentation (https://nixtla.github.io/statsforecast/models.html) but I couldn't locate any method similar to model.get_params() or model.summary() from the sklearn package or any method that would allow me to print the model parameters or a model summary.

Answer: Nixtla statsforecast model stores those information under the hood. There is no method like get_params() to access those, but you can do that pretty easily when you have the model trained. Please see the example below: import pandas as pd from statsforecast import StatsForecast from statsforecast.models import AutoARIMA, AutoCES, AutoETS, AutoTheta train_dt = pd.read_csv('//data_you_will_forecast_in_stastforecast_format.csv') models = [ AutoARIMA(season_length=period), AutoTheta(season_length=period), AutoETS(season_length=period), AutoCES(season_length=period), ] sf = StatsForecast(df=train_df, # used data frame models=models, # a list of models. Select the models you want from models and import them. freq='MS', # a string indicating the frequency of the data. n_jobs=-1, fallback_model=SeasonalNaive(season_length=period) # a model to be used if a model fails. sf.fit(train_df)  When sf models are fit, the data are accessible as follows: sf.fitted_ # access an array of fitted models. sf.fitted_[0][n].model_ # access dictionary of all model's parameters  The dictionary will give all information on the fitted model, including in-sample data, aic (or other metric by which the best model was selected), best model parameters, and many others. In case of AutoARIMA it is under the key 'arma'
Question: I just want to get class data in my python script like: person, car, truck, dog but my output more than this. Also I can not use results as a string. Python script: from ultralytics import YOLO model = YOLO("yolov8n.pt") results = model.predict(source="0")  Output: 0: 480x640 1 person, 1 car, 7.1ms 0: 480x640 1 person, 1 car, 7.2ms 0: 480x640 1 person, 1 car, 7.1ms 0: 480x640 1 person, 1 car, 7.1ms 0: 480x640 1 person, 1 car, 7.1ms 0: 480x640 1 person, 7.9ms 0: 480x640 1 person, 7.1ms 0: 480x640 1 person, 1 car, 7.1ms 0: 480x640 1 person, 1 car, 7.1ms 

Answer: You can pass each class to the model's name dict like this: from ultralytics.yolo.engine.model import YOLO model = YOLO("yolov8n.pt") results = model.predict(stream=True, imgsz=512) # source already setup names = model.names for r in results: for c in r.boxes.cls: print(names[int(c)])  output: YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs bus person person person person image 1/2 /home/xyz/ultralytics/ultralytics/assets/bus.jpg: 512x384 4 persons, 1 bus, 35.7ms person person person tie tie image 2/2 /home/xyz/ultralytics/ultralytics/assets/zidane.jpg: 288x512 3 persons, 2 ties, 199.0ms Speed: 3.9ms pre-process, 117.4ms inference, 27.9ms postprocess per image at shape (1, 3, 512, 512)
Question: I have a lambda that attempts to find out whether a previously executed athena query has returned any rows or not. To do so I am using the boto3 function get_query_runtime_statistics and then extracting the "Rows" data: response = athena_client.get_query_runtime_statistics(QueryExecutionId=query_id) row_count = response["QueryRuntimeStatistics"]["Rows"]["OutputRows"]  However, in a previous execution the response object has not contained the "Rows" data, resulting in a KeyError being thrown. I know I can get around the KeyError by using .get("Rows", {}).get("OutputRows") etc. I reran the exact same query in the athena console (it returns 0 rows) and then used the query ID to get the runtime statistics of this duplicate query execution. This time it had the "Rows" data in the response. Therefore the behaviour doesn't appear to be consistent for a given query string; however, if I get the statistics for the original query execution the response consistently does not contain the "Rows" data. What I want to know is whether every time "Rows" data is not present can I assume that the output row count was zero? I couldn't find anything in the AWS docs explaining why "Rows" may not always be present in the API response. Thanks :) PS. If you don't want to follow the link to the documentation, here is the response schema according to boto3: { 'QueryRuntimeStatistics': { 'Timeline': { 'QueryQueueTimeInMillis': 123, 'QueryPlanningTimeInMillis': 123, 'EngineExecutionTimeInMillis': 123, 'ServiceProcessingTimeInMillis': 123, 'TotalExecutionTimeInMillis': 123 }, 'Rows': { 'InputRows': 123, 'InputBytes': 123, 'OutputBytes': 123, 'OutputRows': 123 }, 'OutputStage': { 'StageId': 123, 'State': 'string', 'OutputBytes': 123, 'OutputRows': 123, 'InputBytes': 123, 'InputRows': 123, 'ExecutionTime': 123, 'QueryStagePlan': { 'Name': 'string', 'Identifier': 'string', 'Children': [ {'... recursive ...'}, ], 'RemoteSources': [ 'string', ] }, 'SubStages': [ {'... recursive ...'}, ] } } } 

Answer: I raised a support ticket and got the follwoing responses:  The query finished successfully but it failed as an async process of getting runtime stats. This is an internal issue and internal team is aware about it and is working on it to fix the same.  I asked for clarification whether this issue only happens on queries that produce zero results, this was the response:  The issue could happen regardless of the query. Also as informed by internal team, it may take approximately 15-30 days to know the root cause and fix the issue.  [sent on 2023-02-10] I hope this is helpful to anyone else who comes across this :)
Question: Is there an update to the library? Before it worked perfectly, and today I updated and it no longer loads I searched but I can't find any other option

Answer: Looks like the new recommendation from the developers is to use a temp-file: https://openpyxl.readthedocs.io/en/3.1/tutorial.html?highlight=save#saving-as-a-stream update: I ended up having to use this with modifications from tempfile import NamedTemporaryFile from openpyxl import Workbook wb = Workbook() with NamedTemporaryFile() as tmp: tmp.close() # with statement opened tmp, close it so wb.save can open it wb.save(tmp.name) with open(tmp.name, 'rb') as f: f.seek(0) # probably not needed anymore new_file_object = f.read()  because the with statement opens the file and then wb.save (which expects a string filename) attempts to open it, resulting in an Exception
Question: I have a python function and want to run it as a separate process with multiprocessing package. def run(ctx: Context): print("hello world!") return ctx  afterward running it as a separate process with the following script: import multiprocessing p = multiprocessing.Process(target=run, args=(ctx, )) p.start() p.join()  Now, I need to capture live stdout and stderr of the above process. Is there any way like as: import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break  But I need to pass the function not running a command with Popen. Do you know how can I read stdout when I run my function in a separate process?

Answer: My approach would be to create a custom context manager that can temporarily replace sys.stdout and sys.stderr with io.String() instances to capture the output and return this. For this you need to make the target of your Process a new function that can setup the context manager and return the results, for which a multiprocessing.Queue is used (this, by the way, would be needed anyway if you expect run to return its result back to the main process): from multiprocessing import Process, Queue from io import StringIO import sys class CaptureOutput: def __enter__(self): self._stdout_output = '' self._stderr_output = '' self._stdout = sys.stdout sys.stdout = StringIO() self._stderr = sys.stderr sys.stderr = StringIO() return self def __exit__(self, *args): self._stdout_output = sys.stdout.getvalue() sys.stdout = self._stdout self._stderr_output = sys.stderr.getvalue() sys.stderr = self._stderr def get_stdout(self): return self._stdout_output def get_stderr(self): return self._stderr_output def run(ctx): print("hello world!") print("It works!", file=sys.stderr) raise Exception('Oh oh!') # Comment out to have a successful completion return ctx def worker(ctx, queue): import traceback with CaptureOutput() as capturer: try: result = run(ctx) except Exception as e: result = e print(traceback.format_exc(), file=sys.stderr) queue.put((result, capturer.get_stdout(), capturer.get_stderr())) if __name__ == '__main__': queue = Queue() ctx = None # for demo purposes p = Process(target=worker, args=(ctx, queue)) p.start() # Must do this call before call to join: result, stdout_output, stderr_output = queue.get() p.join() print('stdout:', stdout_output) print('stderr:', stderr_output)  Prints: stdout: hello world! stderr: It works! Traceback (most recent call last): File "C:\Booboo\test\test.py", line 44, in worker result = run(ctx) File "C:\Booboo\test\test.py", line 36, in run raise Exception('Oh oh!') # Comment out to have a successful completion Exception: Oh oh!
Question: Context I am currently working on a team project, where we need to train neural networks. Some members are working on their local computer, and some on Colab (for GPU usage). We need to have the same dependencies. I am already familiar in using poetry on a local computer, but not on Colab, and I was wondering how to use it in Colab. So I did some tests, and I encountered some issues. Maybe I can find some answers here. Thank you in advance! 😄 Issues 1st issue: poetry add <package> does not update pyproject.toml I want to add a new package, suppose torch. According to poetry's documentation, to install a new package, we need to run poetry add <package>. Since I run this command for the first time, a virtual environment is created, as well as the poetry.lock. But no packages were installed. Moreover, the poetry.lock file is updated, but not the pyproject.toml. This happens only on Colab. I tried on my local computer, and the command indeed automatically update the pyproject.toml file as well. 2nd issue: poetry run pip install <package> does not update pyproject.toml Instead, we can install the package with poetry run pip install <package>. I saw this command on the following GitHub gist. The packages are now installed in the virtual environment, but the pyproject.toml was not updated. Here is a link to the Colab notebook I used for those tests. Thank you again!

Answer: Hello for everyone reading this post. I settled on a solution. The main issue was that pyproject.toml was not updated automatically, so I just decided to modify it by hand. Here is the steps for using poetry in Colab, whether you create your own poetry project, or cloning a repo on Github. https://github.com/elise-chin-adway/poetry-and-colab/blob/main/Using_python_poetry_in_Google_Colab.ipynb I don't know if it is the best solution or not, but I hope it will help other people :)
Question: I am storing paths in a json file using a python script. I want the paths to be stored in the same format (Unix-style) no matter which OS the script is run on. So basically I want to run the os.path.normpath function, but it only converts paths to Unix-style instead of changing its function depending on the host OS. What is the best way to do this?

Answer: You can convert Windows-style path to UNIX-style after calling os.path.normpath. The conversion can be conveniently done with pathlib.PureWindowsPath.as_posix: import os.path from pathlib import PureWindowsPath path = r'\foo\\bar' path = os.path.normpath(path) if os.path.sep == '\\': path = PureWindowsPath(path).as_posix() print(path)  This outputs, in Windows: /foo/bar
Question: I built a Keras sequential model on the simple dataset. I am able to train the model, however every time I try to get a prediction on the same input I get different values. Anyone knows why? I read through different Stackoverflow here (Why the exactly identical keras model predict different results for the same input data in the same env, Keras saved model predicting different values on different session, different prediction after load a model in keras), but couldn't find the answer. I tried to set the Tensorflow seed and still getting different results. Here is my code from pandas import concat from pandas import DataFrame # create sequence length = 10 sequence = [i/float(length) for i in range(length)] # create X/y pairs df = DataFrame(sequence) df = concat([df, df.shift(1)], axis=1) df.dropna(inplace=True) print(df) # convert to LSTM friendly format values = df.values X, y = values[:, 0], values[:, 1] X = X.reshape(len(X), 1, 1) print(X.shape, y.shape)  output is:  0 0 1 0.1 0.0 2 0.2 0.1 3 0.3 0.2 4 0.4 0.3 5 0.5 0.4 6 0.6 0.5 7 0.7 0.6 8 0.8 0.7 9 0.9 0.8 (9, 1, 1) (9,)  Then start building the model #configure network from tensorflow import keras from keras.models import Sequential from keras.layers import Dense from keras.layers import LSTM tf.random.set_seed(1337) n_batch = len(X) n_neurons = 10 #design network model = Sequential() model.add(LSTM(n_neurons, batch_input_shape=(n_batch, X.shape[1], X.shape[2]), stateful=True)) model.add(Dense(1)) model.compile(loss='mean_squared_error', optimizer='adam') model.fit(X,y,epochs=2,batch_size=n_batch,verbose=1,shuffle=False)  Now every time I run the following code to get the prediction I get different results as you can see here model.predict(X) ********output************** array([[0.03817442], [0.07164046], [0.10493257], [0.13797525], [0.17069395], [0.20301574], [0.23486984], [0.26618803], [0.29690543]], dtype=float32)  model.predict(X) ********output************** array([[0.04415776], [0.08242793], [0.12048437], [0.15823033], [0.19556962], [0.2324073 ], [0.26865062], [0.3042098 ], [0.33899906]], dtype=float32) 

Answer: The problem is setting stateful=True in your LSTM layer, as this keeps the state between predict calls, so each prediction depends on previous predictions. So as a solution, set stateful=False.
Question: the following code: import logging print(type(1)) print(type(logging.WARNING))  prints: <class 'int'> <class 'int'>  yet, according to mypy, the first line of this code snippet is legal, but the second is not (Variable "logging.WARNING" is not valid as a type): OneOrTwo = Literal[1,2] # ok WarningOrError = Literal[logging.WARNING, logging.ERROR] # not ok  I do not understand why the definition of OneOrTwo is ok but WarningOrError is not. I would like also know what could be done to use a legal equivalent of WarningOrError, i.e. something I could use like this: def a(arg: WarningOrError)->None: pass  note: mypy redirects to https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases , but this did not clarify things for me.

Answer: Literal only accepts literal arguments. PEP586 has very rigid definitions of what constitutes a "literal" in this context. You can read about it here. The problem with your definition of WarningOrError is that the actual definition of logging.ERROR and WARNING makes them mutable (you can look at the source code of logging here), and hence illegal literals. Constant expressions like 1 or 2, on the other hand, can be known statically (they never change!) and therefore are acceptable literals. Among the accepted legal literals, Enum objects can help you achieve what you want: import logging from typing import Literal from enum import Enum class Log(Enum): ERROR = logging.ERROR WARNING = logging.WARNING LogType = Literal[Log.ERROR, Log.WARNING] def a(arr: LogType): pass
Question: Here's my table:     category number probability     1102 24 0.3   1102 18 0.6   1102 16 0.1   2884 24 0.16   2884 15 0.8   2884 10 0.04     so I want to replace the number column that has probability lower than 15% with the number that has the highest probability within groups:     category number probability     1102 24 0.3   1102 18 0.6   1102 18 0.1   2884 24 0.16   2884 15 0.8   2884 15 0.04    

Answer: Find the number corresponding to max prob in a group then use loc to update values n = df.sort_values('probability').groupby('category')['number'].transform('last') df.loc[df['probability'] <= 0.15, 'number'] = n    category number probability 0 1102 24 0.30 1 1102 18 0.60 2 1102 18 0.10 3 2884 24 0.16 4 2884 15 0.80 5 2884 15 0.04
Question: I'm new to python, I would like to read a column of values ​​from a csv file and add them together, but only those to the left of the "," My csv File: Name Code Angel 19;90 Eduardo 20;21 Miguel 30;45  I would like to be able to sum only the numbers to the left of the "Code" column, so that my output is "19+20+30 = 69". I tried deleting the ";" and converting the string to int but sums but joins the numbers together and I have this output: Your final sum is : 1990 +2021 +3045 = 7056 

Answer: If need sum values before ; use Series.str.extract with casting to integers and then sum: out = df['Code'].str.extract('(.*);', expand=False).astype('int').sum()  Or use Series.str.split with select first values of lists by str[0]: out = df['Code'].str.split(';').str[0].astype('int').sum()  If need sum all values create DataFrame by expand=True and summing first per rows and then Series: out = df['Code'].str.split(';', expand=True).astype('int').sum().sum()  If need sum without ; use Series.str.replace: out = df['Code'].str.replace(';','', regex=True).astype('int').sum()
Question: How can I place specific constraints on the parameters of a Pydantic model? In particular, I would like:  start_date must be at least "2019-01-01" end_date must be greater than start_date code must be one and only one of the values ​​in the set cluster must be one and only one of the values ​​in the set  The code I'm using is as follows: from fastapi import FastAPI from pydantic import BaseModel from typing import Set import uvicorn app = FastAPI() class Query(BaseModel): start_date: str end_date: str code: Set[str] = { "A1", "A2", "A3", "A4", "X1", "X2", "X3", "X4", "X5", "Y1", "Y2", "Y3" } cluster: Set[str] = {"C1", "C2", "C3"} @app.post("/") async def read_table(query: Query): return {"msg": query} if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0", port=8000) 

Answer: Pydantic has a set of constrained types that allows you to define specific constraints on values. start_date must be at least "2019-01-01" >>> class Foo(BaseModel): ... d: condate(ge=datetime.date.fromisoformat('2019-01-01') >>> Foo(d=datetime.date.fromisoformat('2018-01-12')) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pydantic\main.py", line 342, in pydantic.main.BaseModel.__init__ pydantic.error_wrappers.ValidationError: 1 validation error for Foo d ensure this value is greater than or equal to 2019-01-01 (type=value_error.number.not_ge; limit_value=2019-01-01) >>> Foo(d=datetime.date.fromisoformat('2020-01-12')) Foo(d=datetime.date(2020, 1, 12))  end_date must be greater than start_date For more complicated rules, you can use a root validator: from pydantic import BaseModel, root_validator from datetime import date class StartEnd(BaseModel): start: date end: date @root_validator def validate_dates(cls, values): if values['start'] > values['end']: raise ValueError('start is after end') return values StartEnd(start=date.fromisoformat('2023-01-01'), end=date.fromisoformat('2022-01-01'))  Gives: pydantic.error_wrappers.ValidationError: 1 validation error for StartEnd __root__ start is after end (type=value_error)  For code and cluster, you can use an Enum instead from pydantic import BaseModel from enum import Enum # StrEnum in 3.11+ class ClusterEnum(str, Enum): C1 = "C1" C2 = "C2" C3 = "C3" class ClusterVal(BaseModel): cluster: ClusterEnum print(ClusterVal(cluster='C3').cluster.value) # outputs C3