eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have this dataframe df = pl.from_repr(""" ┌─────┬───────┐ │ one ┆ two │ │ --- ┆ --- │ │ str ┆ str │ ╞═════╪═══════╡ │ a ┆ hola │ │ b ┆ world │ └─────┴───────┘ """)  And I want to change hola for hello: shape: (2, 2) ┌─────┬───────┐ │ one ┆ two │ │ --- ┆ --- │ │ str ┆ str │ ╞═════╪═══════╡ │ a ┆ hello │ # <- │ b ┆ world │ └─────┴───────┘  How can I change the values of a row based on a condition in another column? For instance, with PostgreSQL I could do this: UPDATE my_table SET two = 'hello' WHERE one = 'a';  Or in Spark my_table.withColumn("two", when(col("one") == "a", "hello"))  I've tried using with_columns(pl.when(pl.col("one") == "a").then("hello")) but that changes the column "one". EDIT: I could create a SQL instance and plot my way through via SQL but there must be way to achieve this via the Python API.

Answer: You were really close with with_columns(pl.when(pl.col("one") == "a").then("hello")) but you needed to tell it which column that should be. When you don't tell it which column you're referring to then it has to guess and in this case it guessed the column you referred to. Instead you do (df .with_columns( two=pl.when(pl.col('one')=='a') .then(pl.lit('hello')) .otherwise(pl.col('two'))) )  This uses the **kwargs input of with_columns to allow the column named to be on the left of an equal sign as though it were a parameter to a function. You can also use alias syntax like this... (df .with_columns( (pl.when(pl.col('one')=='a') .then(pl.lit('hello')) .otherwise(pl.col('two'))) .alias('two') ) )  Note that I wrapped the entire when/then/otherwise in parenthesis. The order of operations around when/then/otherwise and alias is weird so I find it's best to always completely wrap them in parenthesis to avoid unexpected results. Worst case scenario is you have redundant parenthesis which doesn't hurt anything.
Question: When using backtest and historical_forecast in darts I expect the same error. However, when doing a test, I get different MAPE values for the same input variables. Can somebody explain how this can happen? How can I make the two methods comparable? Example: import pandas as pd from darts import TimeSeries from darts.models import NaiveDrift from darts.metrics import mape df = pd.read_csv('AirPassengers.csv') series = TimeSeries.from_dataframe(df, 'Month', '#Passengers') print("Backtest MAPE: ", NaiveDrift().backtest(series, start=12, forecast_horizon=6, metric=mape)) historical_forecast = NaiveDrift().historical_forecasts(series, start=12, forecast_horizon=6, verbose=False) print("Historical Forecast MAPE: ", mape(historical_forecast, series))  Output: Backtest MAPE: 16.821355933599133 Historical Forecast MAPE: 21.090231183002143 Links Link to the documentation:https://unit8co.github.io/darts/generated_api/darts.models.forecasting.baselines.html Link to the dataset: https://www.kaggle.com/datasets/rakannimer/air-passengers

Answer: The reason the MAPEs are different is because the data used to compute them are different. Historical_forecast() and backtest() have different default values for the parameter "last_points_only". For historical_forecast(), the parameter is set to True, while for backtest() it is False. This means that historical_forecast by default generates one series against which the accuracy is checked. On the other hand, backtest generates multiple series and averages the error among them. The backtest() function has the default "stride" parameter as 1, which means after it makes one prediction, it will move forward one time step and generate another, more forward one step and generate another, and so on. These multiple series will have a different average MAPE than the one series from the historical forecast. If last_points_only is set to True (as it is by default in historical_forecast()), when the function steps forward one time step, it will only include the new (last) time series point in the calculation, instead of the whole series again. You can check this by setting the "last_points_only" parameter in backtest() to True, and you will get the same result for both functions.
Question: In script.py: def f(n, memo={0:0, 1:1}): if n not in memo: memo[n] = sum(f(n - i) for i in [1, 2]) return memo[n] print(f(400))  python3.6 script.py correctly prints f(400), but with python3.7 script.py it stack overflows. The recursion limit is reached at f(501) in 3.6 and at f(334) in 3.7. What changed between Python 3.6 and 3.7 that caused the maximum recursion depth to be exceeded earlier by this code?

Answer: After some git bisecting between Python 3.6.0b1 and Python 3.7.0a1 I found bpo bug #29306 (git commits 7399a05, 620580f), which identified some bugs with the recursion depth counting. Originally, Victor Stinner reported that he was unsure that some new internal API functions for optimised calls (part of the reported call overhead optimisations) were handling the recursion counter properly, but after further discussion it was decided that the problem was more general and that all calls to C functions need to handle the recursion counter too. A simple test script is included in the linked issue that demonstrates that there are issues with recursion counting in older Python versions too; the script depends on a special extension module that is part of the development tree. However, even though Python versions 2.7, 3.5 and 3.6 were shown to be affected the changes were never back-ported. I'm guessing that because those versions didn't receive the call overhead optimisations backporting was going to be a lot of work. I also can’t find an entry for this change in the Python changelog, perhaps because Victor regarded this as part of the optimisation work. The changes mean that sum() and other built-in functions now count against the recursive call limit where they didn’t before.
Question: I use regex pattern to block acronyms while lower casing text. The code is # -*- coding: utf-8 -*- #!/usr/bin/env python from __future__ import unicode_literals import codecs import os import re text = "This sentence contains ADS, NASA and K.A. as acronymns." pattern = r'([A-Z][a-zA-Z]*[A-Z]|(?:[A-Z]\.)+)' matches = re.findall(pattern, text) def lowercase_ignore_matches(match): word = match.group() if word in matches: return word return word.lower() text2 = re.sub(r"\w+", lowercase_ignore_matches, text) print(text) print(text2) matches = re.findall(pattern, text) print (matches)  output is This sentence contains ADS, NASA and K.A. as acronymns. this sentence contains ADS, NASA and k.a. as acronymns. ['ADS', 'NASA', 'K.A.']  The issue is why is it ignoring k.a. while identifying it as acronymns. I wish to retain k.a. as K.A. Kindly help

Answer: The solution with r[\w\.] works in this case but will struggle if the acronym is at the end of a line with a dot after it (i.e. "[...] or ASDF." We use the pattern to identify every acronym, than lowercase the whole string and then replace the acronyms again with their original value. I changed the pattern a bit so that it also supports acronym like "eFUEL" # -*- coding: utf-8 -*- #!/usr/bin/env python from __future__ import unicode_literals import codecs import os import re text = "This sentence contains ADS, NASA and K.A. as acronymns. eFUEL or ASDF." pattern = r'\b([A-z]*?[A-Z](?:\.?[A-Z])+[A-z]*)' # Find all matches of the pattern in the text matches = re.findall(pattern, text) # Make everything lowercase text2 = text.lower() # Replace each match with its original uppercase version for match in matches: text2 = text2.replace(match.lower(), match) print(text2)  The result is: ['ADS', 'NASA', 'K.A', 'eFUEL', 'ASDF'] this sentence contains ADS, NASA and K.A. as acronymns. eFUEL or ASDF.
Question: I am trying to convert this TensorFlow model to onnx. But I get an error message: > python -m tf2onnx.convert --saved-model .\spice --output model.onnx --opset 11 --verbose ... 2023-04-08 18:33:10,811 - ERROR - tf2onnx.tfonnx: Tensorflow op [Real: Real] is not supported 2023-04-08 18:33:10,812 - ERROR - tf2onnx.tfonnx: Tensorflow op [Imag: Imag] is not supported 2023-04-08 18:33:10,879 - ERROR - tf2onnx.tfonnx: Unsupported ops: Counter({'Real': 6, 'Imag': 6}) ... ValueError: make_sure failure: Current implementation of RFFT or FFT only allows ComplexAbs as consumer not {'Real', 'Imag'}  Same happens with the TensorFlow Light (tflight) model: > python -m tf2onnx.convert --opset 16 --tflite .\lite-model_spice_1.tflite --output spice.onnx ... ValueError: make_sure failure: Current implementation of RFFT or FFT only allows ComplexAbs as consumer not {'Imag', 'Real'}  I am on Windows 11, Python 3.10.10, TensorFlow 2.12 This is my first attempt with TensorFlow / ONNX, so I am unsure where the error comes from. Questions  Is it related to TensorFlow, tf2onnx, or the model? Would it work with another setup (maybe on Linux or other TF version)? How to fix the issue? 

Answer:  The error mentioned above occurred because tf2onnx does not support Real and Imag operations. For a list of operations that can be used with tf2onnx, please refer to this link. This will also affect the TensorFlow model. I'm not sure what this means, but as far as I know, it has no relation whatsoever to what is referred to in point number 2. The way to solve this is by changing the Real and Imag operations as follows:  # Compute the real and imaginary components using tf.math.real and tf.math.imag x_real = tf.math.real(x) x_imag = tf.math.imag(x)
Question: The PyTorch DataLoader turns datasets into iterables. I already have a generator which yields data samples that I want to use for training and testing. The reason I use a generator is because the total number of samples is too large to store in memory. I would like to load the samples in batches for training. What is the best way to do this? Can I do it without a custom DataLoader? The PyTorch dataloader doesn't like taking the generator as input. Below is a minimal example of what I want to do, which produces the error "object of type 'generator' has no len()". import torch from torch import nn from torch.utils.data import DataLoader def example_generator(): for i in range(10): yield i BATCH_SIZE = 3 train_dataloader = DataLoader(example_generator(), batch_size = BATCH_SIZE, shuffle=False) print(f"Length of train_dataloader: {len(train_dataloader)} batches of {BATCH_SIZE}")  I am trying to take the data from an iterator and take advantage of the functionality of the PyTorch DataLoader. The example I gave is a minimal example of what I would like to achieve, but it produces an error. Edit: I want to be able to use this function for complex generators in which the len is not known in advance.

Answer: PyTorch's DataLoader actually has official support for an iterable dataset, but it just has to be an instance of a subclass of torch.utils.data.IterableDataset:  An iterable-style dataset is an instance of a subclass of IterableDataset that implements the __iter__() protocol, and represents an iterable over data samples  So your code would be written as: from torch.utils.data import IterableDataset class MyIterableDataset(IterableDataset): def __init__(self, iterable): self.iterable = iterable def __iter__(self): return iter(self.iterable) ... BATCH_SIZE = 3 train_dataloader = DataLoader(MyIterableDataset(example_generator()), batch_size = BATCH_SIZE, shuffle=False)
Question: Pandas 2.0 introduces the option to use PyArrow as the backend rather than NumPy. As of version 2.0, using it seems to require either calling one of the pd.read_xxx() methods with type_backend='pyarrow', or else constructing a DataFrame that's NumPy-backed and then calling .convert_dtypes on it. Is there a more direct way to construct a PyArrow-backed DataFrame?

Answer: If your data are known to be all of a specific type (say, int64[pyarrow]), this is straightforward: import pandas as pd data = {'col_1': [3, 2, 1, 0], 'col_2': [1, 2, 3, 4]} df = pd.DataFrame( data, dtype='int64[pyarrow]', # ... )   If your data are known to be all of the same type but the type is not known, then I don't know of a way to use the constructor. I tried dtype=pd.ArrowDtype, which does not work, and dtype=pd.ArrowDtype(), which needs an argument that I think would have to be a specific dtype.  One option for possibly-mixed and unknown data types is to make a pa.Table (using one of its methods) and then send it to pandas with the types_mapper kwarg. For example, using a dict: import pyarrow as pa data = {'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']} pa_table = pa.Table.from_pydict(data) df = pa_table.to_pandas(types_mapper=pd.ArrowDtype)  The last line is exactly what pd.read_parquet with dtype_backend='pyarrow' does under the hood, after reading parquet into a pa.Table. I thought it was worth highlighting the approach since it wouldn't have occurred to me otherwise. The method pa.Table.from_pydict() will infer the data types. If the data are of mixed type, but known, and speed is very important, see https://stackoverflow.com/a/57939649 for how to make a predefined schema to pass to the pa.Table constructor.  The above method loses most of the flexibility of the DataFrame constructor (specifying an index, accepting various container types as input, etc.). You might be able to code around this and encapsulate it in a function. Another workaround, as mentioned in the question, is to just construct a NumPy-backed DataFrame and call .convert_dtypes on it: import pandas as pd data = {'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']} df = pd.DataFrame( data, index=[4, 5, 6, 7], # ... ).convert_dtypes(dtype_backend='pyarrow')
Question: I have a dataframe which looks like this: label predicted F1 F2 F3 .... F40 major minor 2 1 4 major major 1 0 10 minor patch 4 3 23 major patch 2 1 11 minor minor 0 4 8 patch major 7 3 30 patch minor 8 0 1 patch patch 1 7 11  I have label which is the true label for the id(not shown as it is not relevant), and predicted label, and then set of around 40 features in my df. The idea is to transform these 40 features into 2 dimensions and visualize them true vs predicted. We have 9 cases for all the three labels major,minor and patch vs their predictions. With PCA, it is not able to capture much variance with 2 components and I am not sure how to map the PCA values with the labels and predictions in the original df as a whole. A way to achieve this is to separate all cases into 9 dataframes and achieve the result, but this isn't what I am looking for. Is there any other way I can reduce and visualize the given data? Any suggestions would be highly appreciated.

Answer: You may want to consider a small multiple plot with one scatterplot for each cell of the confusion matrix. If PCA does not work well, t-distributed stochastic neighbor embedding (TSNE) is often a good alternative in my experience. For example, with the iris dataset, which also has three prediction classes, it could look like this: import pandas as pd import seaborn as sns from sklearn.manifold import TSNE iris = sns.load_dataset('iris') # Mock up some predictions. iris['species_pred'] = (40 * ['setosa'] + 5 * ['versicolor'] + 5 * ['virginica'] + 40 * ['versicolor'] + 5 * ['setosa'] + 5 * ['virginica'] + 40 * ['virginica'] + 5 * ['versicolor'] + 5 * ['setosa']) # Show confusion matrix. pd.crosstab(iris.species, iris.species_pred)   species_pred setosa versicolor virginica species setosa 40 5 5 versicolor 5 40 5 virginica 5 5 40  # Reduce features to two dimensions. X = iris.iloc[:, :4].values X_embedded = TSNE(n_components=2, init='random', learning_rate='auto' ).fit_transform(X) iris[['tsne_x', 'tsne_y']] = X_embedded # Plot small multiples, corresponding to confusion matrix. sns.set() g = sns.FacetGrid(iris, row='species', col='species_pred', margin_titles=True) g.map(sns.scatterplot, 'tsne_x', 'tsne_y');
Question: I want to set one field, which cannot be in response model in abstract method. I set this field to private. I want to use this field to validate other public field. class BaseAsset(BaseModel, ABC): amount: int precision: int nai: str _nai_pattern: str = None def __init__(self, **kwargs): super().__init__(**kwargs) object.__setattr__(self, '_nai_pattern', self.set_nai_pattern()) @abstractmethod def set_nai_pattern(self): pass @validator('nai') def check_nai_format(cls, v, values, **kwargs): if v != values['_nai_pattern']: raise ValueError else: return v class Config: underscore_attrs_are_private = True class AssetHive(BaseAsset): def set_nai_pattern(self): return "@@000000021" 

Answer: I would suggest the following approach.  Make nai_pattern a regular (not private) field, but exclude it from dumping by setting exclude=True in its Field constructor. In addition, hook into schema_extra of the model Config to remove the field from the schema as well. Make the method to get the nai_pattern a class method, so that it can be called from inside a validator. Then define a always=True field validator for nai_pattern that checks if the value is None and if so, calls the method to get the value. Write your validator for nai as you did before, but make sure that the nai field itself is defined after nai_pattern because that will ensure the nai validator is called after that of nai_pattern and you will be guaranteed to have a value to check against. ("Validation is done in the order fields are defined.")  Here is my suggested code: from abc import ABC, abstractmethod from typing import Any, Optional from pydantic import BaseModel, Field, validator class BaseAsset(BaseModel, ABC): amount: int precision: int nai_pattern: Optional[str] = Field(default=None, exclude=True) nai: str @classmethod @abstractmethod def get_nai_pattern(cls) -> str: raise NotImplementedError @validator("nai_pattern", always=True) def set_nai_pattern(cls, v: Optional[str]) -> str: if v is None: return cls.get_nai_pattern() return v @validator("nai") def check_nai(cls, v: str, values: dict[str, Any]) -> str: nai_pattern = values.get("nai_pattern") if v != nai_pattern: raise ValueError(f"{v} does not match {nai_pattern=}") return v class Config: @staticmethod def schema_extra(schema: dict[str, Any]) -> None: del schema["properties"]["nai_pattern"]  Demo: from pydantic import ValidationError class Foo(BaseAsset): @classmethod def get_nai_pattern(cls) -> str: return "abc" print(Foo.schema_json(indent=4)) foo = Foo(amount=1, precision=2, nai="abc") print(foo.json(indent=4)) try: Foo(amount=1, precision=2, nai="xyz") except ValidationError as err: print(err.json(indent=4))  Output: { "title": "Foo", "type": "object", "properties": { "amount": { "title": "Amount", "type": "integer" }, "precision": { "title": "Precision", "type": "integer" }, "nai": { "title": "Nai", "type": "string" } }, "required": [ "amount", "precision", "nai" ] }  { "amount": 1, "precision": 2, "nai": "abc" }  [ { "loc": [ "nai" ], "msg": "xyz does not match nai_pattern='abc'", "type": "value_error" } ]
Question: My program should select three cameras and take pictures with each of it. I have the following code at the moment: def getCamera(camera): graph = FilterGraph() print("Camera List: ") print(graph.get_input_devices()) #tbd get right Camera try: device = graph.get_input_devices().index("HD Pro Webcam C920") except ValueError as e: device = graph.get_input_devices().index("Integrated Webcam") return device  The code above worked fine. But I have three similar cameras with the same name. Output of this: graph = FilterGraph() print("Camera List: ") print(graph.get_input_devices())  Is a list with three cameras all the same name. I thought they are in an array and I can select them with this: device = graph.get_input_devices().index(0)  Like any other Array. But I only can access with the Name. Like in the first code example. How can I access the cameras with index?

Answer: You can use the index of the camera in the list to select it. For example, if you want to select the first camera in the list, you can use the following code: device = graph.get_input_devices().index("HD Pro Webcam C920")  To select the second camera: device = graph.get_input_devices().index("HD Pro Webcam C920", 1)  And to select the third camera, you can use: device = graph.get_input_devices().index("HD Pro Webcam C920", 2)  The second argument in the index() method specifies the starting index for the search. You can also modify your getCamera() function to take an argument that specifies the index of the camera you want to use: def getCamera(camera_index): graph = FilterGraph() cameras = graph.get_input_devices() if camera_index >= len(cameras): raise ValueError("Camera index out of range") return cameras[camera_index] for i in range(3): camera = getCamera(i) #Do something with it
Question: I have a dataframe with epoch time. I convert the epoch time to a timestamp with my local timezone. I would like to remove the timezone information but keep my local timezone in the timestamp (subtract the timezone offset from the timestamp and then remove the timezone). This is the code I have: epochs = np.arange(1644516000, 1644516000 + 1800*10, 1800) df = pd.DataFrame({'time': epochs}) df['time'] = pd.to_datetime(df['time'], unit='s').dt.tz_localize("US/Pacific")  I cannot use: dt.tz_localize(None)  Since it converts it back to UTC. My desired output is a timestamp with no timezone information but in my local timezone: pd.date_range('2022-02-10 10:00', freq='30min', periods=10)  How do I do that?

Answer: Essentially you're trying to get whatever time it was locally after x seconds since the unix epoch in a tz-naive timestamp. Achieving this is a bit weird because:  My experience with tz-naive timestamps in pandas says that they are usually "local" to the user's current timezone. Converting to timestamp from an epoch timestamp feels to me a tz-aware operation, given that you count from UTC 1970-01-01, not your local 1970-01-01.  So what I would expect from e.g. pd.to_datetime(1644516000, unit="s") would be one of:  A UTC-localized timestamp: Timestamp('2022-02-10 18:00:00+0000', tz='UTC') The above converted to local: Timestamp('2022-02-10 10:00:00-0800', tz='US/Pacific') Or a tz-naive timestamp representing the local time since UTC epoch: Timestamp('2022-02-10 10:00:00') (which is what you're searching for)  But instead, pd.to_datetime gives you the UTC local time since the UTC epoch, but as a tz-naive timestamp: >>> pd.to_datetime(1644516000, unit="s") Timestamp('2022-02-10 18:00:00')   One solution would be to manually do the three steps above, i.e. by first specifying that the received timestamps are UTC, then converting to your local time, then removing the tz-info: df['time'] = pd.to_datetime( df['time'], unit='s', utc=True ).dt.tz_convert("US/Pacific").dt.tz_localize(None)  but it feels like I'm missing something easier here...