eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Replacing strings by numerical values used to be easy, but since pandas 2.2. the simple approach below throws a warning. What is the "correct" way to do this now? >>> s = pd.Series(["some", "none", "all", "some"]) >>> s.dtypes dtype('O') >>> s.replace({"none": 0, "some": 1, "all": 2}) FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)` 0 1 1 0 2 2 3 1 dtype: int64  If I understand the warning correctly, the object dtype is "downcast" to int64. Perhaps pandas wants me to do this explicitly, but I don't see how I could downcast a string to a numerical type before the replacement happens.

Answer: When you run: s.replace({"none": 0, "some": 1, "all": 2})  The dtype of the output is currently int64, as pandas inferred that the values are all integers. print(s.replace({"none": 0, "some": 1, "all": 2}).dtype) # int64  In a future pandas version this won't happens anymore automatically, the dtype will remain object (you will still have integers but as objects, not int64): pd.set_option('future.no_silent_downcasting', True) print(s.replace({"none": 0, "some": 1, "all": 2}).dtype) # object  You will have to explicitly downcast the objects to integers (after the replacement): s.replace({"none": 0, "some": 1, "all": 2}).infer_objects(copy=False) print(s.replace({"none": 0, "some": 1, "all": 2}) .infer_objects(copy=False).dtype) # int64
Question: How do you express the identity expression in Polars? By this I mean the expression idexpr that when you do lf.filter(idexpr) you get the entirety of lf. Similar to SELECT(*) in SQL. I'm resorting to a logical expression like idexpr = (pl.col("a") == 0) | (pl.col("a") != 0) 

Answer: According to Documentation, what's passed to filter as a predicate needs to be an "Expression(s) that evaluates to a boolean Series." This you already know, since you are passing a logical expression to circumvent it. Easily enough, there's a very simple expression that always evaluates to true: pl.lit(True) or just True. import polars as pl df = pl.DataFrame({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [24, 28, 23], "bool": [True, False, True] }) print(df.filter(pl.lit(True)))  This gives the output of: shape: (3, 4) ┌─────┬─────────┬─────┬───────┐ │ id ┆ name ┆ age ┆ bool │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 ┆ bool │ ╞═════╪═════════╪═════╪═══════╡ │ 1 ┆ Alice ┆ 24 ┆ true │ │ 2 ┆ Bob ┆ 28 ┆ false │ │ 3 ┆ Charlie ┆ 23 ┆ true │ └─────┴─────────┴─────┴───────┘  Edit: Careful though, I found at least two cases where this does not work. Series: This does not seem to work for them at all (polars 1.15): >>> series = pl.Series("A", ["test", "test"]) >>> series.filter(pl.lit(True)) ... AttributeError: 'Expr' object has no attribute '_s'  I assume this is the case because pl.Series.filter works only with a mask anyway. This works however: series.filter([True])  it also works for a dataframe it seems, so if you want to use the same solution for both series and dataframes, this is the one. Null-column: When there's a column of type null (polars 1.12, seems to be fixed in 1.15 though): >>> null_frame = pl.DataFrame({"A": [None, None], "B": ["test", "test"}) >>> null_frame.filter(pl.lit(True)) ... polars.exceptions.ShapeError: filter's length: 1 differs from that of the series: 2
Question: On some web pages, due to rendering issues (e.g., hidden element), WebElement.text may not reveal the underlying text whereas WebElement.get_attribute("textContent") will. Therefore I have written the following utility function: from selenium.webdriver.remote.webelement import WebElement def text(e: WebElement) -> str: return e.text or e.get_attribute("textContent") or "n/a"  WebElement.get_attribute() is deprecated in Selenium 4.27.0. The recommendation is to use WebElement.get_dom_attribute(). However, this is not a drop-in replacement because WebElement.get_dom_attribute() will only reveal attributes declared in the HTML markup. How would I achieve the same functionality without WebElement.get_attribute()?

Answer: Further research reveals that I can use WebElement.get_property() This mean a slight change to the utility function because this function can return any one of str | bool | WebElement | dict. Therefore, it becomes: from selenium.webdriver.remote.webelement import WebElement def text(e: WebElement) -> str | bool | dict | WebElement: return e.text or e.get_property("textContent") or False
Question: In the askcolor function of class Chooser in the pop up you can add color to a custom colors list. I was wondering if I can somehow get the values of those colors, but I could not find a way how to do it. Code for the pop up: from tkinter import colorchooser color_code = colorchooser.askcolor()  Pop up screen shot: 

Answer: The function tkinter.colorchooser.askcolor() doesn't directly provide access to the custom colors list in its popup. The function's main role is to return the selected color. The closest you can get is keeping track of the colors by creating a list (custom_colors = []) and then every time color_code = colorchooser.askcolor() is called, .append to the list. E.g.: from tkinter import colorchooser, Tk, Button root = Tk() custom_colors = [] def choose_color(): color_code = colorchooser.askcolor() if color_code[1]: if color_code[1] not in custom_colors: custom_colors.append(color_code[1]) print(custom_colors) choose_button = Button(root, text="Choose Color", command=choose_color) choose_button.pack(pady=20) root.mainloop()
Question: I'd like to perform the following: Input: df = pl.DataFrame({ "a": [1,15,None,20,None] })  Output: df = pl.DataFrame({ "a": [1,15,None,20,None], "b": [0,14,None,5,None] })  That is, from:    A     1   15   None   20   None    to:    A B     1 0   15 14   None None   20 5   None None    So, what it does:  If the value of "A" is null, then value of B (output column) is also Null If "A" has some value, please retrieve the last Non-Null value in "A", and then subtract the current value in "A" with the previous Non-Null value  I'd like to perform this in python's polars dataframe library, but I can't seem to find a solution. I've tried the following question: How to select the last non-null value from one column and also the value from another column on the same row in Polars? But unfortunately, this does not answer the original problem, since the question performs an aggregation of an entire column, and then takes the last value of that column. What I'd like to do is not to aggregate an entire column, but simply to subtract a current value with a previous non-null value. I have also tried to use rolling: df = df.with_row_index().rolling( index_column = 'index', period = '???i').agg(pl.col("A").last())  But, of course, that does not work because the occurence of Null Values cannot be determined (i.e. it is not periodic, so I don't know how many indexes before the current entry contains a non-null value in "A"). Does anyone knows how to do so? Thanks!

Answer: You can use a combination of shift and forward_fill to get the last non-null value. So with your input, this looks like df = pl.DataFrame({ "a": [1, 15, None, 20, None] }) df.with_columns( # current row value for "a" minus the last non-null value # as the first row has no previous non-null value, # fill it with the first value (1) ( pl.col("a") - pl.col("a").shift(fill_value=pl.col("a").first()).forward_fill() ).alias("b") ) # shape: (5, 2) # ┌──────┬──────┐ # │ a ┆ b │ # │ --- ┆ --- │ # │ i64 ┆ i64 │ # ╞══════╪══════╡ # │ 1 ┆ 0 │ # │ 15 ┆ 14 │ # │ null ┆ null │ # │ 20 ┆ 5 │ # │ null ┆ null │ # └──────┴──────┘
Question: I'm working with Python in an environment where there are some classes which are defined externally (as in, I don't have access to the files where these classes are defined). So I can import these classes and use them, but since VSCode can't resolve the import, there's no autocomplete for them. What I would want: a way to tell VSCode which attributes and methods these classes have, so it can autocomplete for me whenever I use them (same as if I had these classes defined in my workspace and imported normally). Is this possible somehow?

Answer: Autocomplete and IntelliSense are provided for all files within the current working folder in VSCode. They're also available for Python packages that are installed in standard locations. To enable IntelliSense for packages that are installed in non-standard locations, add those locations to the python.autoComplete.extraPaths collection in your settings.json file. Ref:https://code.visualstudio.com/docs/python/settings-reference#_autocomplete-settings
Question: I am generating TOML files with several tables using TOML Kit without any problem in general. So far all the values were either strings or numbers, but today I first bumped into a problem. I was trying to dump a pathlib.Path object and it fails with a ConvertError Unable to convert an object of <class 'pathlib.WindowsPath'> to a TOML item. I fixed right away, adding a str in front, but I was thinking to do something valid in general. Is there a way to teach TOML Kit how to convert a custom object to a valid TOML value? In the case of Path, would be extremely easy.

Answer: You want https://tomlkit.readthedocs.io/en/latest/api/#tomlkit.register_encoder Which you would use like this: from pathlib import Path from typing import Any import tomlkit from tomlkit.items import Item, String, ConvertError class PathItem(String): def unwrap(self) -> Path: return Path(super().unwrap()) def path_encoder(obj: Any) -> Item: if isinstance(obj, Path): return PathItem.from_raw(str(obj)) else: # we cannot convert this, but give other custom converters a # chance to run raise ConvertError tomlkit.register_encoder(path_encoder) obj = {'path': Path('/foo/bar')} document = tomlkit.dumps(obj) assert document == '''\ path = "/foo/bar" '''
Question: I’m working with Odoo (version 15) and trying to implement a login API. I have created a controller that checks the username and password, but it always returns "Invalid credentials" even when I use the correct login information. Here’s my code:  # -*- coding: utf-8 -*- from odoo import http class TestApi(http.Controller): @http.route("/api/check_login", methods=["POST"], type="json", auth="public", csrf=False) def check_login(self, **kwargs): username = kwargs.get('username') password = kwargs.get('password') if username == "admin" and password == "admin": # Replace with actual validation return { "message": "User is logged in.", "username": username } else: return { "error": "Invalid credentials." }  Steps Taken: .I tested the API using Postman with a POST request to http:///api/check_login. .I used the following JSON body:  { "username": "admin", "password": "admin" }  .I confirmed that the credentials work in the Odoo web interface. Questions: What might be causing the API to not recognize valid credentials? Are there any additional configurations I should check? Is there a better method to handle user authentication in Odoo?

Answer: The issue is that the original code expected data in kwargs, which only works for query parameters or form-encoded data. Since the request sent contains a JSON payload in the request body, the username and password were not being retrieved. The modified code resolves this by explicitly parsing the JSON data from the request body using json.loads(), ensuring the credentials are correctly accessed and processed. # -*- coding: utf-8 -*- from odoo import http import json class TestApi(http.Controller): @http.route("/api/check_login", methods=["POST"], type="json", auth="public", csrf=False) def check_login(self): request_data = json.loads(request.httprequest.data) username = request_data.get('username') password = request_data.get('password') if username == "admin" and password == "admin": # Replace with actual validation return { "message": "User is logged in.", "username": username } else: return { "error": "Invalid credentials." }
Question:  Issue 1: The x-ticks (pie pieces) aren't ordered from 0 to 24 (my bad, should be 1) Issue 2: all y-ticks (rings) aren't showing Issue 3: Someone seems to have eaten a part of the polar chart ... I expect to see 31 rings, and 24 "pie pieces".

Answer: import matplotlib.pyplot as plt import numpy as np fig = plt.figure() ax = fig.add_subplot(111, projection='polar') ax.set_xticks(np.arange(1, 25) * np.pi / 12) ax.set_xticklabels([str(number) for number in range(1, 25)]) ax.set_yticks(range(1, 32)) ax.set_yticklabels([str(number) for number in range(1, 32)]) ax.grid(True) ax.scatter(np.pi / 12 * 24, 31) ax.set_ylim(0, 32) plt.show()
Question: The Problem I'm working directly on a Excel sheet with the Python extension for Excel and I'm trying to find out which is the highest number of a list of cells. This is the code for the function that I wrote: def getMax(celle_voti_lista) -> int: valori = [v for v in celle_voti_lista if isinstance(v, (int, float))] return max(valori) if valori else 0  And this is the function call: max(xl("D11:D23"))  But everytime I run this code, It gives me 0 instead of the actual highest number. What should I do?

Answer: try something like this: from openpyxl import load_workbook def xl(range_str, file_path, sheet_name): wb = load_workbook(file_path) sheet = wb[sheet_name] start_cell, end_cell = range_str.split(":") start_row, start_col = int(start_cell[1:]), start_cell[0].upper() end_row, end_col = int(end_cell[1:]), end_cell[0].upper() cells = [] for row in sheet[f"{start_col}{start_row}":f"{end_col}{end_row}"]: for cell in row: cells.append(cell.value) return cells def getMax(celle_voti_lista) -> int: valori = [v for v in celle_voti_lista if isinstance(v, (int, float))] return max(valori) if valori else 0  and then: file_path = "data.xlsx" sheet_name = "Sheet1" max_value = getMax(xl("D11:D23", file_path, sheet_name)) print(max_value)
Question: I want to filter a DataFrame against multiple thresholds, based on the ID's prefix. Ideally I'd configure these thresholds with a dictionary e.g. minimum_thresholds = { 'alpha': 3, 'beta' : 5, 'gamma': 7, 'default': 4 }  For example: data = { 'id': [ 'alpha-164232e7-75c9-4e2e-9bb2-b6ba2449beba', 'alpha-205acbf0-64ba-40ad-a026-cc1c6fc06a6f', 'beta-76ece555-e336-42d8-9f8d-ee92dd90ef19', 'beta-6c91c1cc-1025-4714-a2b2-c30b2717e3c4', 'gamma-f650fd43-03d3-440c-8e14-da18cdeb78d4', 'gamma-a8cb84b5-e94c-46f7-b2c5-135b59dcd1e3', 'pi-8189aff9-ea1c-4e22-bcf4-584821c9dfd6' ], 'freq': [4, 2, 1, 4, 7, 9, 8] }   id freq 0 alpha-164232e7-75c9-4e2e-9bb2-b6ba2449beba 4 1 alpha-205acbf0-64ba-40ad-a026-cc1c6fc06a6f 2 2 beta-76ece555-e336-42d8-9f8d-ee92dd90ef19 1 3 beta-6c91c1cc-1025-4714-a2b2-c30b2717e3c4 4 4 gamma-f650fd43-03d3-440c-8e14-da18cdeb78d4 7 5 gamma-a8cb84b5-e94c-46f7-b2c5-135b59dcd1e3 9 6 pi-8189aff9-ea1c-4e22-bcf4-584821c9dfd6 8  I would then get an output like:  id freq 0 alpha-164232e7-75c9-4e2e-9bb2-b6ba2449beba 4 1 gamma-f650fd43-03d3-440c-8e14-da18cdeb78d4 7 2 gamma-a8cb84b5-e94c-46f7-b2c5-135b59dcd1e3 9 3 pi-8189aff9-ea1c-4e22-bcf4-584821c9dfd6 8  I could do this bluntly by looping through each threshold, but it feels like there must be a more Pythonic way?

Answer: Another possible solution, whose steps are:  First, the id column is split at each hyphen using the str.split method, extracting the first part of each split with str[0].  Then, the resulting first parts are mapped to their corresponding threshold values using the map function, referencing the thresholds dictionary. If a value is not found in thresholds, the default threshold is used.  The freq column is then compared to these threshold values using the ge method, which checks if freq is greater than or equal to the threshold.  Finally, the dataframe is filtered to include only rows where this condition is met.   df[df['freq'] .ge(df['id'].str.split('-').str[0] .map(lambda x: thresholds.get(x, thresholds['default'])))]  Output:  id freq 0 alpha-164232e7-75c9-4e2e-9bb2-b6ba2449beba 4 4 gamma-f650fd43-03d3-440c-8e14-da18cdeb78d4 7 5 gamma-a8cb84b5-e94c-46f7-b2c5-135b59dcd1e3 9 6 pi-8189aff9-ea1c-4e22-bcf4-584821c9dfd6 8
Question: I am getting an error on running cv2.imshow() cv2.imshow("Image", image)  cv2.error: OpenCV(4.9.0) /io/opencv/modules/highgui/src/window.cpp:1272: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvShowImage'  and following suggestions in this forum I did sudo apt install python3-opencv  which installed opencv4.5d. the same code still threw an error and this time I saw the error was coming from opencv4.9. I do not remember installing it, but I found this usr/local/lib/python3.10/dist-packages/opencv_python_headless-4.9.0.80.dist-info /usr/local/lib/python3.10/dist-packages/opencv_python_headless.libs How do I remove 4.9 or make python import 4.5 and not 4.9?? thanks everybody.

Answer: Your problem isn't the way the package was installed. Your problem is that you installed a headless package when you didn't want that.  Run pip3 list. Look for your headless OpenCV. That step is optional but you'll learn something.  Remove the headless package with pip3 uninstall opencv-python-headless.  Install a regular package with pip3 install opencv-python.   Stick with installing OpenCV using pip. Don't use apt to get OpenCV. The package coming from apt is usually stale. Feel free to remove that old OpenCV 4.5 that you got from apt. It'll be something like sudo apt remove python3-opencv. In case you need an older version of OpenCV, you can get that from pip too. Browse the release history of the package, click on the version you need. Up top there's an install command you can copy. Then you can pip3 install opencv-python==4.5.5.64 If you want multiple versions of a pip package side by side, you'll have to start learning about virtual environments.
Question: I have a dataframe using pandas, something like: d = {'X': [1, 2, 3], 'Y': [220, 187, 170]} df = pd.DataFrame(data=d)  the dataframe ends up like    X Y     1 220   2 187   3 170    I can get the y value for an x value of 1.0 using df[df['X'] == 1.0]['Y']  which returns 220 But is there a way to get a linearly interpolated value of Y for an X value between values of X? For example, if I had an X value of 1.5, I would want it to return an interpolated value of 203.5. I tried the interpolate function, but it permanently adjusts the data in the dataframe. I could also write a separate function that would calculate this, but I was wondering if there was a native function in pandas.

Answer: You can use np.interp for this: import numpy as np X_value = 1.5 np.interp(X_value, df['X'], df['Y']) # 203.5  Make sure that df['X'] is monotonically increasing. You can use the left and right parameters to customize the return value for out-of-bounds values: np.interp(0.5, df['X'], df['Y'], left=np.inf, right=-np.inf) # inf # because 0.5 < df['X'].iloc[0]  By default, out-of-bounds values will correspond to the closest valid X value: np.interp(10, df['X'], df['Y']) # 170 # i.e., match for df['X'].iloc[-1]
Question: Postgresql allows NaN values in numeric columns according to its documentation here. When defining Postgres tables using Django ORM, a DecimalField is translated to numeric column in Postgres. Even if you define the column as bellow: from django.db import models # You can insert NaN to this column without any issue numeric_field = models.DecimalField(max_digits=32, decimal_places=8, blank=False, null=False)  Is there a way to use Python/Django syntax to forbid NaN values in this scenario? The Postgres native solution is to probably use some kind of constraint. But is that possible using Django syntax? Edit: As willeM_ Van Onsem pointed out, Django does not allow NaN to be inserted to DecimalField natively. However, the DB is manipulated from other sources as well, hence, the need to have an extra constraint at the DB level (as opposed to Django's built-in application level constraint).

Answer: I don't have a PostgreSQL database to test against but you can try creating a database constraint using a lookup based on the IsNull looukup: from decimal import Decimal from django.db.models import ( CheckConstraint, DecimalField, Field, Model, Q, ) from django.db.models.lookups import ( BuiltinLookup, ) @Field.register_lookup class IsNaN(BuiltinLookup): lookup_name = "isnan" prepare_rhs = False def as_sql(self, compiler, connection): if not isinstance(self.rhs, bool): raise ValueError( "The QuerySet value for an isnan lookup must be True or False." ) sql, params = self.process_lhs(compiler, connection) if self.rhs: return "%s = 'NaN'" % sql, params else: return "%s <> 'NaN'" % sql, params class Item(Model): numeric_field = DecimalField( max_digits=32, decimal_places=8, blank=False, null=False, ) class Meta: constraints = [ CheckConstraint( check=Q(numeric_field__isnan=False), name="numeric_field_not_isnan", ), ]
Question: I have a model from hugging face and would like to use it for performing word comparisons. At first I thought of performing a series of similarity calculations across words of interest but quickly I found that this problem would exponentially grow as the number of words expanded as well. A solution I thought about is plotting a skip gram where all words result on a 2 dimensional plane and then can simply perform clustering on the coordinates to find similar words. The problem here is that this requires a bert model and a low embedding layer that can be mapped. As I have a pretrained model, I don't know if I can create a skip gram with from it. I was hoping to calculate the embedding and through the use of a transformation, convert the embedding into coordinates that I can plot myself. I though do not know if this is possible or reasonable I tried to do it though with the code below from sklearn.manifold import TSNE from transformers import AutoModel, AutoTokenizer # target word word = ["Slartibartfast"] # model setup model = 'Alibaba-NLP/gte-multilingual-base' tokenizer = AutoTokenizer.from_pretrained(model) auto_model = AutoModel.from_pretrained(model, trust_remote_code=True) # embbed and calculate batch_dict = self.tokenizer(text_list, max_length=8192, padding=True, truncation=True, return_tensors='pt') result = auto_model(**batch_dict) embeddings = outputs.last_hidden_state[:, 0][:768] # transform to coordinates clayer = TSNE(n_components=3, learning_rate='auto', init='random', perplexity=50) embedding_numpy = embeddings.detach().numpy() clayer.fit_transform(embedding_numpy) # crashes here saying perplexity must be less than n_samples 

Answer: After more through reading, it was brough to my attention that it would be impossible to use TSNE in the manner which I was hoping as the dimensions generated by TSNE is only representative of the training data. Further fitting with new data or transformation of data not within the training set would result in outputs that are not on a similar range and thus noncomparable. I found a replacement to TSNE which is called umap. umap is also for dimension reduction but it can be fitted multiple times and data can be transformed along the same range. I will explore umap and see if it will work for what I need.