eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe that looks roughly like this:     timestamp mmsi departures_count calc_speed coursechange     2012-01-31 07:11:59 1.252340e+12 1 4.041755 0.000000   2012-01-31 07:22:20 1.252340e+12 1 5.209561 30.000000   2012-01-31 07:35:34 1.252340e+12 2 5.766184 1.000000   2012-01-31 07:45:35 1.252340e+12 2 5.932638 4.000000   2016-11-24 17:05:21 2.775153e+14 1 1.673716 17.000000   2016-11-24 17:21:21 2.775153e+14 1 0.725156 180.800003   2016-11-24 17:38:40 2.775153e+14 1 0.418093 117.500003     The Dataframe has more columns and way more rows (2284331 rows x 16 columns), but these are the important ones needed for the operations. It consists of unique identifiers mmsi and departures_count indicating an associated trip. Since the timestep between datapoints varies significantly, I am trying to resample the data for better comparison. I would like the timesteps to always be 10 minutes. This means that I want to upsample the data if there are multiple datapoints within a 10 minute span und downsample it (using interpolation) when there is a larger timegap. This needs to be done for each departures_count per mmsi. I tried using the following but it either just returns NaN's or the same value for each row of a trip (departure). timeindex = pd.to_datetime(df['timestamp'], format='%Y-%m-%d %H:%M:%S') df.index = timeindex group = df.groupby(['mmsi', 'departures_count']) df_test = group[['coursechange', 'calc_speed']].resample('10Min').interpolate(method='linear')  If possible, I would also like to change the interpolation method to cubic instead of linear. Any suggestions would be highly appreciated!

Answer: You can try: def fn(g): rng = pd.date_range( g.index[0].floor("10min"), g.index[-1].ceil("10min"), freq="10min" ) g = ( g.reindex(g.index.union(rng)) .interpolate(method="linear", limit_direction="both") .reindex(rng) ) return g df["timestamp"] = pd.to_datetime(df["timestamp"]) df = df.set_index("timestamp") out = df.groupby(["mmsi", "departures_count"], group_keys=False).apply(fn) print(out)  Prints:  mmsi departures_count calc_speed coursechange 2012-01-31 07:10:00 1.252340e+12 1.0 4.041755 0.000000 2012-01-31 07:20:00 1.252340e+12 1.0 4.625658 15.000000 2012-01-31 07:30:00 1.252340e+12 1.0 5.209561 30.000000 2012-01-31 07:30:00 1.252340e+12 2.0 5.766184 1.000000 2012-01-31 07:40:00 1.252340e+12 2.0 5.849411 2.500000 2012-01-31 07:50:00 1.252340e+12 2.0 5.932638 4.000000 2016-11-24 17:00:00 2.775153e+14 1.0 1.673716 17.000000 2016-11-24 17:10:00 2.775153e+14 1.0 1.357529 71.600001 2016-11-24 17:20:00 2.775153e+14 1.0 1.041343 126.200002 2016-11-24 17:30:00 2.775153e+14 1.0 0.571624 149.150003 2016-11-24 17:40:00 2.775153e+14 1.0 0.418093 117.500003
Question: When coding in Python using Replit, I get the annoying warning "Line too long" (pyright-extended).  How can I get Replit to always ignore displaying this type of warnings?

Answer: Just add 'E501' to the "[tool.ruff]" section of the "pyproject.toml"-file, to disable the warning in the default python template.  https://ask.replit.com/t/python-88-characters-maximum-in-a-line/74941
Question: I wonder if I have designed my script correctly (it works so far) because of a loop of warnings from which I cannot get out. I am trying to define a function that either is able to parse a JSON response from a website or it simply terminates the program. Here my code: from requests import get, Response, JSONDecodeError from typing import Any from sys import exit def logout() -> None: exit() def get_page(url:str) -> dict[Any,Any]: try: response: Response = get(url) page: dict[Any,Any] = response.json() except JSONDecodeError as e: print(e) logout() return page try: result: dict[Any, Any] = get_page("https://stackoverflow.com/") except Exception as e: #handle exception pass print(result['something'])  Pylance informs me that page (as in return page) and result (as in print(result['something']) is possibly unbound. As far as I understand there are different possible workarounds to this problem, but none of them solves entirely the problem:  When moving return page before the except statement, mypy raises a Missing return statement. If I ignore the "error" from mypy, Pylance informs me that I should anyway sign my function specifying that it should return dict[Any,Any] | None. Changing the return signature to dict[Any,Any] means that outside the function result should be result: dict[Any,Any]|None but then result['something'] is possibly unbound and Object of type "None" is not subscriptable initializing page = {} at the beginning of the function is a possible solution, but what if page is a custom class which need some parameter that I cannot provide? I could raise an Exception from within the function if there would have been a way to inform Pylance about this possible behaviour (is there?)  My point is: the function will NEVER return None right? So, am I doing something wrong? How do you deal with this? Should I refactor my function?

Answer: typing.NoReturn is used for functions that never return. For example: from typing import NoReturn def stop() -> NoReturn: raise RuntimeError('no way')  So you should type hint the return value of logout() with NoReturn: from typing import NoReturn def logout() -> NoReturn: exit()
Question: I am building a Python package using setuptools. This is the basic structure: /path/to/project/ ├── myproj/ │ ├── __init__.py │ └── my_module.py ├── pyproject.toml ├── setup.cfg └── ## other stuff  This is the content of pyproject.toml [build-system] requires = [ "setuptools", "setuptools-scm"] build-backend = "setuptools.build_meta"  and this is the content of setup.cfg [project] name = "my_package" author = 'my name' requires-python = ">=3.8" keywords = ["one", "two"] license = {text = "BSD-3-Clause"} classifiers = [] dependencies = [] [metadata] version = attr: my_package.__version__ [options] zip_safe = True packages = my_package include_package_data = True install_requires = matplotlib==3.6.0 spacy==3.7.2 networkx>=2.6.3 nltk>=3.7 shapely>=1.8.4 pandas>=1.3.5 community>=1.0.0b1 python-louvain>=0.16 numpy==1.23.4 scipy [options.package_data] my_package = ## pointers to data [options.packages.find] exclude = examples* demos*  I can run successfully python -m build. Then, if I pip install it locally, i.e., pointing directly to the .tar file that was built, I can install it. However, if I upload it to test.pypi and then I pip install it from test.pypi, I get the following error: pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> [3 lines of output] Looking in indexes: https://test.pypi.org/simple/ ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none) ERROR: No matching distribution found for setuptools [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip.  Looks like an issue with setuptools, but I really cannot figure out how to fix it. Any suggestions?

Answer:  However, if I upload it to test.pypi and then I pip install it from test.pypi  I assume this means you're using --index-url https://test.pypi.org/simple/, which is confirmed by the build output Looking in indexes: https://test.pypi.org/simple/. That tells pip to use TestPyPI exclusively as the package index. You will only be able to install packages that are published to TestPyPI, which setuptools is not. If you want to install packages from both TestPyPI and PyPI during the same pip install command, you can use --extra-index-url instead of index-url. Note that this means that any package can be installed from either TestPyPI or PyPI, with no guarantees as to which packages come from which index. If you want install particular packages from particular indices (e.g. my_package from TestPyPI, and everything else from PyPI), you can try the solutions from Install specific package from specific index.
Question: I think after the recent update of Polars (version 0.20.7), it stopped supporting the use of map_groups with with_columns I got a Type Error stating TypeError: cannot call `map_groups` when grouping by an expression  I am trying to Group By a column called Product Number and fill the null values using backward strategy for this Group By. The code I wrote was. df1 = ( df1 .group_by(['Product Number']) .map_groups(lambda df1: ( df1 .with_columns( df1['New Date'].fill_null(strategy='backward').alias('New Date1') ) ) ) )  Is there alternate solution which supports grouping with with_columns ?

Answer: Instead, you can use pl.Expr.over to compute expressions over certain groups. ( df1 .with_columns( pl.col("New Date") .fill_null(strategy="backward") .over("'Product Number'") .alias("New Date1") ) )
Question: I'd like to have Pydantic fields that are discriminated based on a class variable. For example: from pydantic import BaseModel, Field from typing import Literal, ClassVar class Cat(BaseModel): animal_type: ClassVar[Literal['cat']] = 'cat' class Dog(BaseModel): animal_type: ClassVar[Literal['dog']] = 'dog' class PetCarrier(BaseModel): contains: Cat | Dog = Field(discriminator='animal_type')  But this code throws an exception at import time:  pydantic.errors.PydanticUserError: Model 'Cat' needs a discriminator field for key 'animal_type'  If I remove the ClassVar annotations, it works fine, but then animal_type is only available as an instance property, which is less convenient in my case. Can anyone help me use class attributes as discriminators with Pydantic? This is Pydantic version 2.

Answer: This is a very interesting question. Broadly it seems to me that Pydantic does not support discriminated unions for ClassVar literals, but that asks the question, why not? An important thing to note about ClassVar's, is they don't become regular fields in pydantic (https://docs.pydantic.dev/2.3/usage/models/#class-vars). In some way they're quite a lot like PrivateAttr. They're only accessible via get, they don't feature in models. If we consider your question again but using PrivateAttr, it makes sense why this isn't supported: from pydantic import BaseModel, Field, PrivateAttr from typing import Literal class Cat(BaseModel): _animal_type: Literal['cat'] = PrivateAttr(default='cat') class Dog(BaseModel): _animal_type: Literal['dog'] = PrivateAttr(default='dog') class PetCarrier(BaseModel): contains: Cat | Dog = Field(discriminator='_animal_type')  Field cannot access the discriminator _animal_type because it isn't accessible to it - it's a PrivateAttr. A discriminator has to be something that is a part of the model - even if instances aren't part of your use case, you have to consider what would happen if someone did make an instance. Just like PrivateAttr, ClassVar is a field that's not really part of the model, so it's impossible to use as a discriminator. In Pydantic I would think of ClassVar as kind of like a PrivateAttr that's publically accessible and can't be altered.
Question: how do i fix the error shown in the pic, i was trying to edit my VSCode settings.json, installed the extension Ruff and tried to edit the settings based on an article I've read here: https://medium.com/@ordinaryindustries/the-ultimate-vs-code-setup-for-python-538026b34d94 but i get the error shown in the pic below 

Answer: Smart Tips(Ctrl+Space) displays these two settings with five values, which are：  "always": Triggers Code Actions on explicit saves and auto saves triggered by window or focus changes. "explicit": Triggers Code Actions only when explicitly saved "never": Never triggers Code Actions on save "false": Never triggers Code Actions on save. This value will be deprecated in favor of "never". "true": Triggers Code Actions only when explicitly saved. This value will be deprecated in favor of "explicit". From the explanation, it is clear that false and true are about to be deprecated, so please use the corresponding "never" or "explicit".
Question: in The context of the sphinxcontrib organisation the package are supposed to all live in a sphinxcontrib namespace package like "sphinxcontrib.icon" or "sphinxcontrib.badge". The file have the following structure: sphinxcontrib-icon/ ├── sphinxcontrib/ │ └── icon/ │ └── __init__.py └── pytproject.toml  In a setuptools based pyproject.toml file I would do the following: # pyproject.toml [build-system] requires = ["setuptools>=61.2", "wheel", "pynpm>=0.2.0"] build-backend = "setuptools.build_meta" [tool.setuptools] include-package-data = false packages = ["sphinxcontrib.icon"]  Now I would like to migrate to hatchings but I don't manage to reprduce this behaviour. In my parameters I do: # pyproject.toml [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["sphinxcontrib/skeleton"]  In the "site-packages" folder my lib is not under in "sphincontrib/icon" as I would expect. How should I adapt my code to make it work ? I have a example sitting here: https://github.com/sphinx-contrib/sphinxcontrib-skeleton/tree/test Building the docs with nox -s docs is failing because the package is not really installed.

Answer: copying here the answer given by the hatchling repository maintainers:  The packages option is for the case where your code lives under a directory like src/ and will collapse until the last path component. So if your namespace package lived under such a directory then you could set that to src/sphinxcontrib but since it's not you case  So I could simply do: [tool.hatch.build.targets.wheel] only-include = ["sphinxcontrib"]  Note that you could also use the regular include option there but that would still have to recurse other directories, so just a performance hack.
Question: I use nearly the same code as here in this GitHub repo to get embeddings from OpenAI: oai = OpenAI( # This is the default and can be omitted api_key="sk-.....", ) def get_embedding(text_to_embed, openai): response = openai.embeddings.create( model= "text-embedding-ada-002", input=[text_to_embed] ) return response embedding_raw = get_embedding(text,oai)  According to the GitHub repo, the vector should be in response['data'][0]['embedding']. But it isn't in my case. When I printed the response variable, I got this: print(embedding_raw)  Output: CreateEmbeddingResponse(data=[Embedding(embedding=[0.009792150929570198, -0.01779201813042164, 0.011846082285046577, -0.0036859565880149603, -0.0013213189085945487, 0.00037509595858864486,..... -0.0121011883020401, -0.015751168131828308], index=0, object='embedding')], model='text-embedding-ada-002', object='list', usage=Usage(prompt_tokens=360, total_tokens=360))  How can I access the embedding vector?

Answer: Simply return just the embedding vector as follows: def get_embedding(text_to_embed, openai): response = openai.embeddings.create( model= "text-embedding-ada-002", input=[text_to_embed] ) return response.data[0].embedding # Change this embedding_raw = get_embedding(text,oai)
Question: I'm trying to use a dataclass as a (more strongly typed) dictionary in my application, and found this strange behavior when using a custom type subclassing list within the dataclass. I'm using Python 3.11.3 on Windows. from dataclasses import dataclass, asdict class CustomFloatList(list): def __init__(self, args): for i, arg in enumerate(args): assert isinstance(arg, float), f"Expected index {i} to be a float, but it's a {type(arg).__name__}" super().__init__(args) @classmethod def from_list(cls, l: list[float]): return cls(l) @dataclass class Poc: x: CustomFloatList p = Poc(x=CustomFloatList.from_list([3.0])) print(p) # Prints Poc(x=[3.0]) print(p.x) # Prints [3.0] print(asdict(p)) # Prints {'x': []}  This does not occur if I use a regular list[float], but I'm using a custom class here to enforce some runtime constraints. How do I do this correctly? I'm open to just using .__dict__ directly, but I thought asdict() was the more "official" way to handle this A simple modification makes the code behave as expected, but is slightly less efficient: from dataclasses import dataclass, asdict class CustomFloatList(list): def __init__(self, args): dup_args = list(args) for i, arg in enumerate(dup_args): assert isinstance(arg, float), f"Expected index {i} to be a float, but it's a {type(arg).__name__}" super().__init__(dup_args) @classmethod def from_list(cls, l: list[float]): return cls(l) @dataclass class Poc: x: CustomFloatList p = Poc(x=CustomFloatList.from_list([3.0])) print(p) print(p.x) print(asdict(p)) 

Answer: If you look at the source code of asdict, you'll see that passes a generator expression that recursively calls itself on the elements of a list when it encounters a list:  elif isinstance(obj, (list, tuple)): # Assume we can create an object of this type by passing in a # generator (which is not true for namedtuples, handled # above). return type(obj)(_asdict_inner(v, dict_factory) for v in obj)  But your implementation depletes any iterator it gets in __init__ before the super call. Don't do that. You'll have to "cache" the values if you want to use the superclass constructor. Something like: class CustomFloatList(list): def __init__(self, args): args = list(args) for i, arg in enumerate(args): assert isinstance(arg, float), f"Expected index {i} to be a float, but it's a {type(arg).__name__}" super().__init__(args)  Or perhaps: class CustomFloatList(list): def __init__(self, args): super().__init__(args) for i, arg in enumerate(self): if not isinstance(arg, float): raise TypeError(f"Expected index {i} to be a float, but it's a {type(arg).__name__}")
Question: I've put my data to the pytorch tensor and now i want to split into the batches size 64. I got the following code: batch = 0 BATCH_SIZE = 64 X_train = x_scaled.to(device) y_train = y_scaled.to(device) for batch in range(0,len(X_train[0]),BATCH_SIZE): ### Training model.train() # train mode is on by default after construction # 1. Forward pass y_pred = model(X_train[0:2][batch:batch+BATCH_SIZE])  The shape of the tensor is: torch.Size([2, 11938]). And i want to slice it into [2,64]. However it do not slice properly and gives an error: mat1 and mat2 shapes cannot be multiplied (2x11938 and 2x64) What i want: tensor([[0.0000, 0.0002, 0.0004, 0.0005, 0.0007, 0.0009, 0.0011, 0.0013, 0.0014, 0.0016, 0.0018, 0.0018, 0.0020, 0.0022, 0.0023, 0.0025, 0.0027, 0.0029, 0.0029, 0.0031, 0.0032, 0.0034, 0.0036, 0.0038, 0.0040, 0.0041, 0.0043, 0.0045, 0.0047, 0.0049, 0.0051, 0.0052, 0.0054, 0.0056, 0.0058, 0.0060, 0.0061, 0.0061, 0.0063, 0.0065, 0.0067, 0.0069, 0.0070, 0.0072, 0.0074, 0.0076, 0.0078, 0.0079, 0.0081, 0.0083, 0.0083, 0.0085, 0.0087, 0.0088, 0.0090, 0.0092, 0.0094, 0.0094, 0.0096, 0.0097, 0.0099, 0.0101, 0.0103, 0.0105],[0.0684, 0.0684, 0.0684, 0.0684, 0.0684, 0.0684, 0.0684, 0.0684, 0.0684, 0.0703, 0.0703, 0.0703, 0.0684, 0.0684, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0703, 0.0712, 0.0712, 0.0712, 0.0712, 0.0712, 0.0712, 0.0712, 0.0712]], device='cuda:0', dtype=torch.float64)  What i get: tensor([[0.0000e+00, 1.8038e-04, 3.6076e-04, ..., 9.9964e-01, 9.9982e-01, 1.0000e+00], [6.8395e-02, 6.8395e-02, 6.8395e-02, ..., 5.7695e-01, 5.7695e-01, 5.7695e-01]], device='cuda:0', dtype=torch.float64)  How can i slice the torch tensor to the requiered shape?

Answer: You should read how indexing works in pytorch/numpy and other similar libraries. You have a tensor of shape (2, 11938). When you index with X_train[0:2], you get a tensor of size (2, 11938). You don't need to index if you want all the rows. When you index with X_train[0:2][batch:batch+BATCH_SIZE], you're indexing the result of X_train[0:2] with [batch:batch+BATCH_SIZE]. This means you are still indexing the first axis. If you want a chunk of size BATCH_SIZE from the second axis, you need to index the second axis, ie X_train[:, batch:batch+BATCH_SIZE].
Question: Suppose I have the following Polars DataFame: import polars as pl df = pl.DataFrame({ 'lst': [[0, 1], [9, 8]], 'val': [3, 4] })  And I want to add the number in the val column, to every element in the corresponding list in the lst column, to get the following result: ┌───────────┬─────┐ │ lst ┆ val │ │ --- ┆ --- │ │ list[i64] ┆ i64 │ ╞═══════════╪═════╡ │ [3, 4] ┆ 3 │ │ [13, 12] ┆ 4 │ └───────────┴─────┘  I know how to add a constant value, e.g.: new_df = df.with_columns( pl.col('lst').list.eval(pl.element() + 2) )  But when I try: new_df = df.with_columns( pl.col('lst').list.eval(pl.element() + pl.col('val')) )  I get the following error: polars.exceptions.ComputeError: named columns are not allowed in `list.eval`; consider using `element` or `col("")`  Is there any elegant way to achieve my goal (without map_elements)? Thanks in advance.

Answer: If lst has a fixed number of items for the whole column AND you know how many in advance then you can do this: df.with_columns( pl.concat_list(pl.col('lst').list.get(x) + pl.col('val') for x in range(2)) )  If not, then you can still do this: df.with_columns( pl.concat_list( pl.col('lst').list.get(x) + pl.col('val') for x in range(df['lst'].list.len().max()) ) .list.gather( pl.int_ranges(0,pl.col('lst').list.len()) ) )  The way this works is that it replaces the 2 from the first case with df['lst'].list.len().max() which is the longest list in the whole column. Then it does a gather so that it only takes the first n elements that correspond to the length of each particular lst. concat_list isn't super efficient so it might be the case that explode, group_by is more efficient. (df .with_row_index('i') .explode('lst') .group_by('i',maintain_order=True) .agg( pl.col('lst')+pl.col('val'), pl.col('val').first(), ) .drop('i') )
Question: I have response from server: { "date": "2024-02-05 15:34:44", "status": True, "data": [ [1, "red"], [2, "blue"], [3, "yellow"] ] }  And i want to serialize this response to pydantic model, but i don't know how parse list (example [1, "red"]) in pydantic model class Item(BaseModel): # how convert list from data to this model id: int color: str ... class Model(BaseModel): date: datetime status: bool data: list[Item] 

Answer: You can work for example with a model_validator to parse the list into the Item model: from pydantic import BaseModel, model_validator data = { "date": "2024-02-05 15:34:44", "status": True, "data": [ [1, "red"], [2, "blue"], [3, "yellow"] ] } class Item(BaseModel): id: int color: str @model_validator(mode="before") @classmethod def validate(cls, values): id, color = values return {"id": id, "color": color} class Model(BaseModel): date: datetime status: bool data: list[Item] print(Model(**data))  Which prints: date=datetime.datetime(2024, 2, 5, 15, 34, 44) status=True data=[Item(id=1, color='red'), Item(id=2, color='blue'), Item(id=3, color='yellow')]  There are likely other approaches, such as AliasPath, but this one seems the simplest. I hope this helps!