eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a Polars DataFrame, and I want to highlight the top 3 values for each column using the style and loc features in Polars. I can achieve this for individual columns, but my current approach involves a lot of repetition, which is not scalable to many variables. import polars as pl import polars.selectors as cs from great_tables import loc, style df = pl.DataFrame({ "id": [1, 2, 3, 4, 5], "variable1": [15, 25, 5, 10, 20], "variable2": [40, 30, 50, 10, 20], "variable3": [400, 100, 300, 200, 500] }) top3_var1 = pl.col("variable1").is_in(pl.col("variable1").top_k(3)) top3_var2 = pl.col("variable2").is_in(pl.col("variable2").top_k(3)) ( df .style .tab_style( style.text(weight="bold"), loc.body("variable1", top3_var1) ) .tab_style( style.text(weight="bold"), loc.body("variable2", top3_var2) ) )  This works, but it's not scalable for many columns since I have to manually define top3_var for each column. I’ve tried using pl.all().top_k(3) to make the process more automatic: ( df .style .tab_style( style.text(weight="bold", ), loc.body("variable1", top3_var1) ) .tab_style( style.text(weight="bold"), loc.body("variable2", top3_var2) ) )  However, I’m not sure how to apply the style and loc methods to highlight only the top 3 values in each column individually without affecting the entire row.

Answer: Update. Since the writing of my original answer, a mask parameter was added to great_tables.loc.body providing a native solution to the problem. import polars as pl import polars.selectors as cs from great_tables import loc, style df = pl.DataFrame({ "id": [1, 2, 3, 4, 5], "variable1": [15, 25, 5, 10, 20], "variable2": [40, 30, 50, 10, 20], "variable3": [400, 100, 300, 200, 500] }) ( df .style .tab_style( style.text(weight="bold"), locations=loc.body( mask=cs.exclude("id").is_in(cs.exclude("id").top_k(3)) ) ) )    Original. As outlined in the comments, there are already some discussions on GitHub regarding adding a loc.body(mask=...) argument suitable for the use-case. Until this feature is implemented, you could create a GT (Great Table) object and iteratively use gt.tab_style as follows. This avoids the manual chaining of tab_style calls. import polars as pl import polars.selectors as cs from great_tables import GT, loc, style df = pl.DataFrame({ "id": [1, 2, 3, 4, 5], "variable1": [15, 25, 5, 10, 20], "variable2": [40, 30, 50, 10, 20], "variable3": [400, 100, 300, 200, 500] }) gt = GT(df) for col in df.select(cs.exclude("id")).columns: gt = gt.tab_style( style.text(weight="bold"), loc.body(col, pl.col(col).is_in(pl.col(col).top_k(3))) ) gt
Question: When using a dictionary to occasionally replace values, are .get(x, x) and .get(x) or x equivalent? For example: def current_brand(brand_name): rebrands = { "Marathon": "Snickers", "Opal Fruits": "Starburst", "Jif": "Cif", "Thomson": "TUI", } return rebrands.get(brand_name, brand_name) # or return rebrands.get(brand_name) or brand_name # this is forbidden - cannot use `default` keyword argument here return rebrands.get(brand_name, default=brand_name) assert current_brand("Jif") == "Cif" assert current_brand("Boots") == "Boots"  I think .get(x) or x is clearer, but that's pretty much a matter of opinion, so I'm curious if there's a technical benefit or drawback to one or the other I've not spotted. Edit: sorry, to be clear, I'm assuming that the dictionary does not contain falsey values as that would't make sense in this context (i.e. in the example above you're not recording that "Somerfield" rebranded as "")

Answer: In answer to your original question: or will short-circuit if your value is Falsey, so there are plenty of values where the two statements will behave differently. my_dict = { 'foo': 0, 'bar': "", 'baz': [], } x = 'foo' print(repr(my_dict.get(x, x))) print(repr(my_dict.get(x) or x))  In answer to your edit, if you can guarantee that your dict is definitely a dict, and that the values in your dict are always Truthy, then the two should behave equivalently. There may be some performance tradeoffs either way, but I generally don't bother to get hung up on that. If this code ends up in a hot loop, then profile it, write the faster version, then stick a comment nearby explaining that your code's weird because the weird way is faster. In my opinion the downside of your method is philosophical. To my eyes, .get(x, ...) communicates your intent as "retrieve key x, and if it's not there, default to ...". .get(x) or ... communicates "if it's not there, or is defined as a Falsey value, default to ...". The intent displayed in your question suggests you should use .get(x, x), but for a simple example like this it hardly matters.
Question: I am trying to install detectron2 on ubuntu and face a weird python dependency problem. In short - pytorch is installed (with pip), torchvision is installed (with pip), but when I run pip install 'git+https://github.com/facebookresearch/detectron2.git'  I get error ModuleNotFoundError: No module named 'torch' as for dependencies (detectron2_test) ubuntu@LAPTOP:~$ pip install torchvision Requirement already satisfied: torchvision in ./detectron2_test/lib/python3.12/site-packages (0.19.1+cu118) Requirement already satisfied: numpy in ./detectron2_test/lib/python3.12/site-packages (from torchvision) (1.26.3) Requirement already satisfied: torch==2.4.1 in ./detectron2_test/lib/python3.12/site-packages (from torchvision) (2.4.1+cu118) (...) (detectron2_test) ubuntu@LAPTOP:~$ which pip /home/ubuntu/detectron2_test/bin/pip (detectron2_test) ubuntu@LAPTOP:~$ which python /home/ubuntu/detectron2_test/bin/python (detectron2_test) ubuntu@LAPTOP:~$ which python3 /home/ubuntu/detectron2_test/bin/python3  Any suggestions are appreciated!

Answer: This is probably due to the isolation mechanism of the pip building process. Basically, the installation requires torch to be installed to work, but recent versions of pip use some isolation that does not allow the build process to access installed packages. You can disable that isolation by using this command: $ pip install --no-build-isolation 'git+https://github.com/facebookresearch/detectron2.git'  This happens a lot for packages that need torch, probably because they tend to verify torch version and also import it to check for cuda and/or other capabilities, or to compile some kernels.
Question: I have an Firebase Cloud Functions codebase that uses Firestore database. Everything below works when I use it in local emulator using firebase emulators:start but when I need to deploy it to Firebase I got below error: Error: User code failed to load. Cannot determine backend specification main.py import json from firebase_functions import https_fn from firebase_admin import initialize_app, firestore import flask from enum import Enum from flask import g from endpoints.moon_phase import moon_phase_bp # Initialize Firebase app and Firestore initialize_app() db = firestore.client() app = flask.Flask(__name__) # Set up a before_request function to make db available in blueprints @app.before_request def before_request(): # g.db = db print("before_request") app.register_blueprint(moon_phase_bp) //Doesn't even use db, but in the future it will. # Firebase Function to handle requests @https_fn.on_request() def astro(req: https_fn.Request) -> https_fn.Response: with app.request_context(req.environ): return app.full_dispatch_request()  If I update the db initialisation to db = firestore.client it deploys, but obviously, it's a function reference, so I cannot use Firestore db in my endpoints. This also means it's not related to my Firebase credentials or project setup. What might be the issue here?

Answer: Figured out the below answer in the Firebase repo fixes the issue:  I've discovered if I move the db = firestore.client() into my cloud function, I'm able to deploy.  https://github.com/firebase/firebase-functions-python/issues/126#issuecomment-1682542027 It's really weird that Google does not update the docs or fixes the issue, but for now, this answer unblocks me, hope it helps others too.
Question: I want to save train dataset, test dataset, and validation dataset in 3 separate folders. Doing this for training and testing is easy # Get training and testing data all_training_data = getattr(datasets, config['Pytorch_Dataset']['dataset'])( root= path_to_data + "/data_all_train_" + config['Pytorch_Dataset']['dataset'], train=True, download=True, # If present it will not download the data again transform=ToTensor() ) test_data = getattr(datasets, config['Pytorch_Dataset']['dataset'])( root= path_to_data + "/data_test_" + config['Pytorch_Dataset']['dataset'], train=False, download=True, # If present it will not download the data again transform=ToTensor() )  This code makes use of torchvision.datasets to load and save to disk the dataset specified in config['Pytorch_Dataset']['dataset'] (e.g. MNIST). However there is no option to load a validation set this way, there is no validation=True option. I could split the train dataset into train and validation with torch.utils.data.random.split, but there are two main problems with this approach:  I don't want to save the folder data_all_train, I want to save only 2 folders, one with the true training part and one with the validation part I would like PyTorch to understand if data_train and data_validation are present, and in this case it should not download again data_all_train, even if not present 

Answer: You don't have to save the split results on separate folders to maintain reproducibility, which is what I am assuming you really care for. You could instead fix the seed before calling split like this: torch.manual_seed(42) data_train, data_val = torch.utils.data.random_split(data_all_train, (0.7, 0.3))  Then you get to maintain just the initial folders while also ensuring the train and val splits are consistent across trials.  But the caveat to the above is, you are fixing the global seed so you are also losing the randomness you might desire in the dataloader shuffling and such, which will end up identical per trial. To avoid that, you can narrow down the scope you are fixing the seed by setting it only for the generator you pass to the split call: split_gen = torch.Generator() split_gen.manual_seed(42) data_train, data_val = torch.utils.data.random_split( data_all_train, (0.7, 0.3), generator=split_gen)
Question: I have VS Code set up with Python extension, but without Jupyter extension. VS Code lets you send Python code from the editor to what it calls the "Native REPL" (a Jupyter-type interface without full notebook capabilities) or the "Terminal REPL" (the good old Python >>> prompt). See https://code.visualstudio.com/docs/python/run. I prefer working in the Terminal REPL. I have my keyboard shortcuts configured to use Ctrl+Enter (instead of Shift+Enter) to send the selection to the terminal: [ { "key": "ctrl+enter", "command": "python.execSelectionInTerminal", "when": "editorTextFocus && !findInputFocussed && !isCompositeNotebook && !jupyter.ownsSelection && !notebookEditorFocused && !replaceInputFocussed && editorLangId == 'python'" }, { "key": "shift+enter", "command": "-python.execSelectionInTerminal", "when": "editorTextFocus && !findInputFocussed && !isCompositeNotebook && !jupyter.ownsSelection && !notebookEditorFocused && !replaceInputFocussed && editorLangId == 'python'" } ]  On Linux, this works as expected. Editor: print("Hello, World!")  What happens in terminal: >>> print("Hello, world!") Hello, world!  On Windows, the same actions (select line, hit Ctrl+Enter) produces the following: >>> KeyboardInterrupt >>> print("Hello, world!") Hello, world!  This might be some harmless little visual noise, but it seems to be interfering with plotting. When I initialize a plot with: import matplotlib.pyplot as plt fig, ax = plt.subplots()  print("Hello, world!") now generates the following: >>> print("Hello, world!") Traceback (most recent call last): File "<stdin>", line 0, in <module> KeyboardInterrupt  Note that putting code directly into the terminal (instead of using VS Code's send-to-terminal feature) still functions normally. How can I fix the send-to-terminal feature?

Answer: Roll back your python extension to 14.0 and disable auto-update will fix the issue. Reference: https://github.com/microsoft/vscode-python/issues/24251
Question: I am making a python application that uses 4 ttk Frames within its main window. The first two frames should expand both vertically and horizontally to fill available space. Frames 3 and 4 should only expand horizontally and be a fixed height vertically. This is my code so far (minimum working example): import tkinter as tk from tkinter import ttk tk_root = tk.Tk() tk_root.geometry('500x500') frame_1 = ttk.Frame(tk_root, padding=10, borderwidth=3, relief=tk.GROOVE) frame_2 = ttk.Frame(tk_root, padding=10, borderwidth=3, relief=tk.GROOVE) frame_3 = ttk.Frame(tk_root, padding=10, borderwidth=3, relief=tk.GROOVE, height=50) frame_4 = ttk.Frame(tk_root, padding=10, borderwidth=3, relief=tk.GROOVE, height=50) frame_1.pack(fill=tk.BOTH, expand=True) frame_2.pack(fill=tk.BOTH, expand=True) frame_3.pack(fill=tk.X, expand=True) frame_4.pack(fill=tk.X, expand=True) tk_root.mainloop()  When i run this the first two frames only expand to 50% of the window height and empty space is left around frames 3 and 4 (screenshot). I would like to have frames 3 and 4 be a constant height while frames 1 and 2 expand to fill all available space above frames 3 and 4. 

Answer: acw1668 gave the answer in a comment:  Remove expand=True for frame 3 and 4.  That did the trick.
Question: Update: This was fixed by pull/20417 in Polars 1.18.0  I'm using .map_elements to apply a complex Python function to every element of a polars series. This is a toy example: import polars as pl df = pl.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}) def sum_cols(row): return row["A"] + row["B"] df.with_columns( pl.struct(pl.all()) .map_elements(sum_cols, return_dtype=pl.Int32).alias("summed") ) shape: (3, 3) ┌─────┬─────┬────────┐ │ A ┆ B ┆ summed │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i32 │ ╞═════╪═════╪════════╡ │ 1 ┆ 4 ┆ 5 │ │ 2 ┆ 5 ┆ 7 │ │ 3 ┆ 6 ┆ 9 │ └─────┴─────┴────────┘  However, when my function raises an exception, Polars silently uses Nulls as the output of the computation: def sum_cols(row): raise Exception return row["A"] + row["B"] df.with_columns( pl.struct(pl.all()) .map_elements(sum_cols, return_dtype=pl.Int32).alias("summed") ) shape: (3, 3) ┌─────┬─────┬────────┐ │ A ┆ B ┆ summed │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i32 │ ╞═════╪═════╪════════╡ │ 1 ┆ 4 ┆ null │ │ 2 ┆ 5 ┆ null │ │ 3 ┆ 6 ┆ null │ └─────┴─────┴────────┘  How can I make the Polars command fail when my function raises an exception?

Answer: I'm pretty sure this is a bug in Polars.  https://github.com/pola-rs/polars/issues/19315 https://github.com/pola-rs/polars/issues/14821  As a workaround, you could use .map_batches() to pass the whole "column" instead: import polars as pl df = pl.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}) def sum_cols(col): raise Exception return pl.Series(row["A"] + row["B"] for row in col) df.with_columns( pl.struct(pl.all()).map_batches(sum_cols) )  Which propagates exceptions as one would expect. # ComputeError: Exception:
Question: I'm using ale in vim, and I want to add ruff as fixer for python. So, in .vimrc, I added: let g:ale_fixers = { \ 'python': ['ruff'], \ 'javascript': ['eslint'], \ 'typescript': ['eslint', 'tsserver', 'typecheck'], \}  Then, when executing in vim the command ALEFix, I've go this error:  117: Unknown function: ale#fixers#ruff#Fix  Do you know how to integrate ruff as fixer in vim with ALE ?

Answer: https://docs.astral.sh/ruff/editors/setup/#vim , open "With the ALE plugin for Vim or Neovim." The docs shows " Linter let g:ale_linters = { "python": ["ruff"] } " Formatter let g:ale_fixers = { "python": ["ruff_format"] }
Question: I have a piece of code that works in Polars 0.20.19, but I don't know how to make it work in Polars 1.10. The working code (in Polars 0.20.19) is very similar to the following: def format_all_string_fields_polars() -> pl.Expr: return ( pl.when( (pl.col(pl.Utf8).str.strip().str.lengths() == 0) | # ERROR ON THIS LINE (pl.col(pl.Utf8) == "NULL") ) .then(None) .otherwise(pl.col(pl.Utf8).str.strip()) .keep_name() ) df.with_columns(format_all_string_fields_polars())  I have converted the pl.Utf8 dtype to pl.String, but it keeps giving me the same error:  AttributeError: 'ExprStringNameSpace' object has no attribute 'strip'  The function is supposed to perform the When-Then operation on all string fields of the dataframe, in-place, but return all columns in the dataframe (including the non-string columns as well). How do I convert this function to a working piece of code in Polars 1.10?

Answer:  pl.Utf8 was renamed to pl.String .str.strip() was renamed to .str.strip_chars() .str.lengths() was split into .str.len_chars() and .str.len_bytes() .keep_name() was renamed to .name.keep()  def format_all_string_fields_polars() -> pl.Expr: return ( pl.when( (pl.col(pl.String).str.strip_chars().str.len_chars() == 0) | (pl.col(pl.String) == "NULL") ) .then(None) .otherwise(pl.col(pl.String).str.strip_chars()) .name.keep() )
Question:  How would I loop through HTML Table Rows in Python? Just to let y'all know, I am working on the website: https://schools.texastribune.org/districts/. What I'm trying to do is click each link in the table body (?) and extract the total number of students:  What I have so far: response = requests.get("https://schools.texastribune.org/districts/") soup = BeautifulSoup(response.text) data = [] for a in soup.find_all('a', {'class': 'table table-striped'}): response = requests.get(a.get('href')) asoup = BeautifulSoup(response.text) data.append({ 'url': a.get('href'), 'title': a.h2.get_text(strip=True), 'content': asoup.article.get_text(strip=True) }) pd.DataFrame(data)  This is my first ever time web scraping something.

Answer: You should not have class_="td" when finding the <td> elements, they don't have any class. There's no <ul> elements in the table, so view = match.find('ul',class_="tr") won't find anything. You need to find the <a> element, gets its href, and load that to get the total students. d = {} for match in soup.find_all('td'): link = match.find("a") if link and link.href: school_page = requests.get("https://schools.texastribune.org" + link.href) school_soup = BeautifulSoup(school_page, 'lxml') total_div = school_soup.find("div", class_="metric", text="Total students" if total_div: amount = total_div.find("p", class_="metric-value") d[link.text] = amount.text print(d)
Question: Given a dataframe like the one below, how do I filter for the earlest and latest date in each month? Note the actual data runs to tens of thousands of rows. Input:    Date Deg     02/01/1990 1210.92   13/01/1990 1226.83   14/01/1990 1224.52   15/01/1990 1220.77   08/02/1990 1164.32   09/02/1990 1156.72   12/02/1990 1145.18   13/02/1990 1146.88   24/02/1990 1149.07    Desired output:    Date Deg     02/01/1990 1210.92   15/01/1990 1220.77   08/02/1990 1164.32   24/02/1990 1149.07   

Answer: Your data looks sorted. Try this: df["year"] = df["Date"].dt.year df["month"] = df["Date"].dt.month return pd.concat( [ df.groupby(["month", "year"]).last(), df.groupby(["month", "year"]).first() ] ).reset_index(drop=True).sort_values(by="Date")
Question: Consider the following example:  UserId = NewType('UserId', int) ProductId = NewType('ProductId', int)  But I can also do, the following: UserId: TypeAlias = int ProductId: TypeAlias = int  So why should I use NewType over TypeAlias or vice versa? Are they both interchangeable?

Answer: Aliases don't distinguish between each other. New types do. Consider this example: Meter = NewType('Meter', float) Gram = NewType('Gram', float) MeterSquared = NewType('MeterSquared', float) def area(length: Meter, width: Meter) -> MeterSquared: return MeterSquared(length * width)  Now area(3, 5) won't type check, nor would area(Meter(3), Gram(5)), but area(Meter(3), Meter(5)) will. If you had defined Meter and Gram as mere type aliases, all three would type check. Basically, a NewType is, well, a new type entirely distinct from the "base" type, just with the same underlying representation. A type alias, on the other hand, is just another name for an existing type, and the two are entirely interchangeable.
Question: I have two arrays containing 60 0's or 1's. One is defined as result and the other is defined as daysinfected. The goal is to look at each element of result and set that element to -1 IF it is > 0 AND IF the corresponding element in the daysinfected element is 0. By printing result for debugging after this code it is obvious that it is not generating -1 values where expected (I.e., where these conditions are both met). In fact it doesn't seem to be modifying anything in result. for i in range(len(result)): for i in range (len(daysinfected)): if result[i] > 0 and daysinfected[i] == 0: i in result == -1 

Answer: Your outer loop is redundant here, just this is sufficient: for i in range (len(daysinfected)): if result[i] > 0 and daysinfected[i] == 0: i in result == -1  Now, in the 3rd line, i in result == -1 this is an expression, not an assignment. its interpreted like so: (i in result) == -1 where i in result is checking if i exists in result, then you're checking if its equal to -1 with ... == -1, not assigning a value. So really, you want this: for i in range (len(daysinfected)): if result[i] > 0 and daysinfected[i] == 0: result[i] = -1 # assign -1 to the i-th value in result, notice the single =  However, with numpy arrays, you should try to avoid looping in python as much as possible. Generally, numpy provides utility functions which can almost always do the trick much faster and are much more readable: result = np.where((result > 0) & (daysinfected == 0), -1, result)
Question: I'm on python 3.10. I'm using PyCharm's default type checker and MyPy. Here is the protocol I defined: class OnSubscribeFunc(Protocol): def __call__(self, instrument: str, *, x: int) -> AsyncGenerator: ...  When create a method that implements it like this: class A: async def subscribe(self, instrument: str, *, x: int): yield ... a: OnSubscribeFunc = A().subscribe # this apparently is where it gets it wrong  I get this warning: Expected type 'OnSubscribeFunc', got '(instrument: str, Any, x: int) -> AsyncGenerator' instead If I remove the * from my implementation however, the warning disappears. I would expect it to be the other way around because not having the * allows the implementation to have non-keyword-only arguments which might not be what I'm aiming for with my protocol. So for comparison - this implementation gives no warning: class A: async def subscribe(self, instrument: str, x: int): yield ...  This does not make any sense to me, why does it behave like this and is this expected or is it a bug in my type checker?

Answer: This is a known bug of PyCharm. Mypy and Pyright both accept your code as it is. Put a # type: ignore or # noqa there and move on.
Question: I mentioned the condition to print at particular places but instead of printing at those locations code just appends "+" to "-" even with range condition it exceeds the limit. I wanted to print "+" at 0, 8, 16, 24 index locations and print "-" in between them. def display(board): for row in range(25): print("-", end = "") if row == 0 or row == 8 or row == 16 or row == 24: print("+", end = "")  The outcome when I invoke the function is: -+--------+--------+--------+  I tried to modify code so that "+" stays in condition loop when "-" is outside but output is different def display(board): for row in range(25): if row == 0 or row == 8 or row == 16 or row == 24: print("+", end = "") print("-", end = "")  +--------+--------+--------+-  My expected outcome is supposed to be: +-------+-------+-------+ 

Answer: You've just got to use an if else to only print 1 char per iteration of your loop. Without the else, you are printing twice on each multiple of 8. def display(board): for row in range(25): if row == 0 or row == 8 or row == 16 or row == 24: print("+", end = "") else print("-", end = "")  You could also simplify the code and make it a bit more readable using some math. def display(board): for row in range(25): if row % 8 == 0: print("+", end = "") else print("-", end = "")