eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have been given a problem to solve for my assignment. Here's the description for my problem: In the cell below, you have a DataFrame df that consists of 10 columns of floating-point numbers. Exactly 5 entries in each row are NaN values. For each row of the DataFrame, find the column which contains the third NaN value. You should return a Series of column labels: e, c, d, h, d nan = np.nan data = [[0.04, nan, nan, 0.25, nan, 0.43, 0.71, 0.51, nan, nan], [ nan, nan, nan, 0.04, 0.76, nan, nan, 0.67, 0.76, 0.16], [ nan, nan, 0.5 , nan, 0.31, 0.4 , nan, nan, 0.24, 0.01], [0.49, nan, nan, 0.62, 0.73, 0.26, 0.85, nan, nan, nan], [ nan, nan, 0.41, nan, 0.05, nan, 0.61, nan, 0.48, 0.68]] columns = list('abcdefghij') df = pd.DataFrame(data, columns=columns) # write a solution to the question here  This is my solution: result = df.apply(lambda row: row.isna().idxmax(), axis=1) print(result)  My code returns b, a, a, a, b, a, whereas the desired output is e, c, d, h, d. My solution returns the index of the first NaN but according to the question I want the third NaN in each row. How can i do that? Or is there an alternate solution that gives the desired output.

Answer: idxmax was a good approach, you can combine this with a mask that indicates the 3rd NaN, for this use cumsum: m = df.isna() out = (m & m.cumsum(axis=1).eq(3)).idxmax(axis=1)  since idxmax always returns the first value if there are several maxes, this could be simplified to: out = df.isna().cumsum(axis=1).eq(3).idxmax(axis=1)  Output: 0 e 1 c 2 d 3 h 4 d dtype: object  Intermediates: # m a b c d e f g h i j 0 False True True False True False False False True True 1 True True True False False True True False False False 2 True True False True False False True True False False 3 False True True False False False False True True True 4 True True False True False True False True False False # m.cumsum(axis=1) a b c d e f g h i j 0 0 1 2 2 3 3 3 3 4 5 1 1 2 3 3 3 4 5 5 5 5 2 1 2 2 3 3 3 4 5 5 5 3 0 1 2 2 2 2 2 3 4 5 4 1 2 2 3 3 4 4 5 5 5 # m.cumsum(axis=1).eq(3) a b c d e f g h i j 0 False False False False True True True True False False 1 False False True True True False False False False False 2 False False False True True True False False False False 3 False False False False False False False True False False 4 False False False True True False False False False False # m & m.cumsum(axis=1).eq(3) a b c d e f g h i j 0 False False False False True False False False False False 1 False False True False False False False False False False 2 False False False True False False False False False False 3 False False False False False False False True False False 4 False False False True False False False False False False  Alternatively, if you can't assume a specific number of NaNs in each row, using melt + groupby.nth: N = 3 out = (df.melt(ignore_index=False) # reshape to long .loc[lambda x: x['value'].isna()] # only keep NaNs .groupby(level=0)['variable'].nth(N-1) # keep 3rd row per group )  Output: 1 c 2 d 4 d 0 e 3 h Name: variable, dtype: object
Question: Just to be clear, this question has nothing to do with the regular expression itself and my code is perfectly running even though it is not passing mypy strict verification. Let's start from the basic, I have a class defined as follows: from __future__ import annotations import re from typing import AnyStr class MyClass: def __init__(self, regexp: AnyStr | re.Pattern[AnyStr]) -> None: if not isinstance(regexp, re.Pattern): regexp = re.compile(regexp) self._regexp: re.Pattern[str] | re.Pattern[bytes]= regexp  The user can build the class either passing a compiled re pattern or AnyStr. I want the class to store in the private _regexp attribute the compiled value. So I check if the user does not provided a compiled pattern, then I compile it and assign it to the private attribute. So far so good, even though I would have expected self._regexp to be type re.Pattern[AnyStr] instead of the union of the type pattern types. Anyhow, up to here everything is ok with mypy. Now, in some (or most) cases, the user provides the regexp string via a configuration TOML file, that is read in, parsed in a dictionary. For this case I have a class method constructor defined as follow:  @classmethod def from_dict(cls, d: dict[str, str]) -> MyClass: r = d.get('regexp') if r is None: raise KeyError('missing regexp') return cls(regexp=r)  The type of dictionary will be dict[str, str]. I have to check that the dictionary contains the right key to prevent a NoneType in case the get function cannot find it. I get the error:  error: Argument "regexp" to "MyClass" has incompatible type "str"; expected "AnyStr | Pattern[AnyStr]" [arg-type]  That looks bizarre, because str should be compatible with AnyStr. Let's say that I modify the dictionary typing to dict[str, AnyStr]. Instead of fixing the problem, it multiplies it because I get two errors:  error: Argument "regexp" to "MyClass" has incompatible type "str"; expected "AnyStr | Pattern[AnyStr]" [arg-type]  error: Argument "regexp" to "MyClass" has incompatible type "bytes"; expected "AnyStr | Pattern[AnyStr]" [arg-type]  It looks like I am in a loop: when I think I have fixed something, I just moved the problem back elsewhere.

Answer:  AnyStr is a type variable, and type variables should either appear 2+ times in a function signature or 1+ time in the signature and 1 time in the enclosing class as a type variable. If you have neither of these situations, you'd be better off to use a union. See mypy Playground  — comment by dROOOze
Question: I have generated a matplotlib figure that I want to save to a PDF. So far, this is straightforward. import matplotlib.pyplot as plt x = [1, 2, 3, 4] y = [3, 5, 4, 7] plt.scatter(x, y) plt.savefig( "example.pdf", bbox_inches = "tight" ) plt.close()  However, I would like the figure to appear in the middle of a standard page and have some white space around the figure, rather than fill up the entire page. How can I tweak the above to achieve this?

Answer: Another option is to use constrained layout and set a rectangle that you want the all the plot elements to be contained in. import matplotlib.pyplot as plt x = [1, 2, 3, 4] y = [3, 5, 4, 7] fig, ax = plt.subplots(figsize=(8.3, 11.7), layout='constrained') fig.get_layout_engine().set(rect=(0.15, 0.3, 0.7, 0.4)) # left, bottom, width, height in fractions of the figure ax.scatter(x, y) fig.savefig('example.pdf') plt.close()
Question: I want to create an ORC file compressed with ZLIB compression level 9. Thing is, when using pyarrow.orc, I can only choose between "Speed" and "Compression" mode, and can't control the compression level E.g. orc.write_table(table, '{0}_zlib.orc'.format(file_without_ext), compression='ZLIB', compression_strategy='COMPRESSION')  Ideally I'm looking for a non existing compression_level parameter, any help would be appreciated.

Answer: The Apache ORC library (which is used internally by other libraries for ORC support) doesn't allow to set the compression level freely (neither the C++ nor the Java implementation). The C++ library supports only CompressionStrategy_SPEED and CompressionStrategy_COMPRESSION (source): enum CompressionStrategy { CompressionStrategy_SPEED = 0, CompressionStrategy_COMPRESSION };  The Java library offers an additional FASTEST option (source): enum SpeedModifier { /* speed/compression tradeoffs */ FASTEST, FAST, DEFAULT }  There is an open request in the project about this: Support maximum compression ratio in setSpeed. It was created a year ago but the feature has not been implemented so far. So, unless you patch the library yourself, there is no way to set a high compression level.
Question: I wrote an algorithm that uses a long list. Since numpy.array should perform better in dealing with long data, I wrote another version using numpy.array. list version: import math def PrimeTable(n:int) -> list[int]: nb2m1 = n//2 - 1 l = [True]*nb2m1 for i in range(1,(math.isqrt(n)+1)//2): if l[i-1]: for j in range(i*3,nb2m1,i*2+1): l[j] = False return [2] + [i for i,v in zip(range(3,n,2), l) if v]  array version: import math, numpy def PrimeTable2(n:int) -> list[int]: nb2m1 = n//2 - 1 l = numpy.full(nb2m1, True) for i in range(1,(math.isqrt(n)+1)//2): if l[i-1]: for j in range(i*3,nb2m1,i*2+1): l[j] = False return [2] + [i for i,v in zip(range(3,n,2), l) if v]  It turns out that numpy.array version is 1x slower than list version. The graph:where y0 is PrimeTable and y1 is PrimeTable2 Why is numpy.array so slow, and how can I improve it?

Answer: Numpy is optimized for vectorized operations, where large chunks of data are processed in bulk. Instead of updating element-by-element(l[j] = False) we can use slicing to update a range of values at once. Also reducing the number of python loops should make the code more effective. I've further optimised the code by using a single boolean array. The sieve tracks only odd numbers directly. This eliminates the need for multiplication and division steps for odd indices during iteration. After the sieve is complete, the prime numbers are directly reconstructed from the indices, avoiding intermediate computations. Try out this updated code. This should work better than the list version of code: import numpy as np def PrimeTable2_optimized_v2(n: int) -> list[int]: if n < 2: return [] if n == 2: return [2] # Only consider odd numbers; even numbers > 2 are not prime sieve = np.ones(n // 2, dtype=bool) limit = int(n**0.5) + 1 for i in range(1, limit // 2 + 1): if sieve[i]: start = 2 * i * (i + 1) sieve[start::2 * i + 1] = False # Convert sieve to primes primes = 2 * np.nonzero(sieve)[0] + 1 primes[0] = 2 return primes.tolist()
Question: I am using plotnine with a date x-axis plot and want to put x-axis date values on top of the chart but couldn't find a way to do it. I have seen in ggplot it can be done using scale_x_discrete(position = "top")  but with scale_x_datetime() I couldn't find any position parameter in plotnine. Below is the sample code: import pandas as pd from plotnine import * # Create a sample dataset data = pd.DataFrame({ 'date': pd.date_range('2022-01-01', '2022-12-31',freq="M"), 'value': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120] }) # Create the line chart (ggplot(data, aes(x='date', y='value')) + geom_line() + labs(title='Line Chart with Dates on X-Axis', x='Date', y='Value') + theme_classic() )  Would really appreciate any help !! Update (facet plot) after the answer: # sample dataset new_data = { 'date': pd.date_range('2022-01-01', periods=8, freq="ME"), 'parent_category': ['Electronics', 'Electronics', 'Fashion', 'Fashion', 'Home Goods', 'Electronics', 'Fashion','Electronics'], 'child_category': ['Smartphones', 'Laptops', 'Shirts', 'Pants', 'Kitchenware','Laptops', 'Shirts', 'Smartphones'] } # Create DataFrame new_data = pd.DataFrame(new_data) new_data  plot with answer tips (ggplot(new_data, aes(x="date", y="child_category", group="child_category")) + geom_line(size=1, color="pink") + geom_point(size=3, color="grey") + facet_wrap("parent_category", ncol=1, scales="free_y") + theme_538() + theme(axis_text_x=element_text(angle=45, hjust=1), panel_grid_major=element_blank(), figure_size=(8, 6), axis_line_x=element_line(position=('axes',1))) )   In facet plot it doesn't go to top of the chart.

Answer: Try this: # Create the line chart (ggplot(data, aes(x='date', y='value')) + geom_line() + labs(title='Line Chart with Dates on X-Axis', x='Date', y='Value') + theme_classic() + theme(axis_line_x=element_line(position=('axes', 1))) )  Here's the source code explanation/implementation for position=(axes, 1) for more clarity. Update: For the sake of completeness here's more detail on the facet plot for the updated question (all credit goes to @ViSa for figuring this out in the comments): According to the source code of plotnine for facet_wrap:  scales : Whether x or y scales should be allowed (free) to vary according to the data on each of the panel.  And the allowed values are: "fixed", "free", "free_x", "free_y" with the default value being "fixed". So, setting scales to free_x or free should resolve the problem shown in the image shared in the question. facet_wrap("parent_category", ncol=1, scales="free_x")  Here's the updated image with this change:
Question: I am looking to extract rows from my dataset based on a pattern like condition. The condition I'm looking for is finding periods in a battery's charging history where it discharged from 100-0% without charging in between. For example, in this dataset below I would be interested in a function that would only return timestamp of 7 to 12 as it contains a full discharge of the battery.    timestamp Charge level (%)     1 50   2 55   3 40   4 60   5 80   6 100   7 100   8 85   9 60   10 55   11 40   12 0   13 20    The approach I have tried is to use the loc function in Pandas to look for rows with a charge level of 0% and then backtrack until I reach a row with a charge level of 100%. But I am struggling with the backtracking part in this approach.

Answer: [Code updated according to comments] The idea I use is to keep only the rows with 0 and 100 and the final rows of interest will be the ones with 100 followed by 0. [after checking with .diff(1) that the values are monotonically decreasing] I also updated your example to include some more difficult cases like when it start discharging and then start charging before it was fully discharged [or fully discharged]. with pd.option_context('display.max_columns', None): display(df.T) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 level 20 0 40 60 80 100 100 85 60 40 40 0 100 20 20 55 100 10 100 0 100 20 40 0 20 df["diff1"] = df["level"].diff(1) df.set_index("time", inplace=True) res = df.copy() mask =((res["level"] == 100) | (res["level"] == 0)) res = res.loc[mask] mask1 = (res["level"] == 100) & (res["level"].shift(-1) == 0) mask2 = (res["level"] == 0) & (res["level"].shift(1) == 100) res = res.loc[mask1 | mask2, ["level"]] level time 7 100 12 0 19 100 20 0 21 100 24 0 # Get the start and end of all the segments having levels [100, ..., 0] t_start, t_end = res[res["level"] == 100].index, res[res["level"] == 0].index # Among all the segments keep only the ones with level monotonically decreasing idx = [(t_start[i], t_end[i]) for i in range(len(t_start)) if (df.loc[t_start[i]+1:t_end[i], "diff1"] <= 0).all()] print(idx) # All resulting groups with (start_time, end_time) [(7, 12), (19, 20)] # Example to get the sum of all the levels in result group n°0 (idx[0] ie (7, 12)) gp0 = df.loc[idx[0][0]:idx[0][1], ["level"]] display(gp0) level time 7 100 8 85 9 60 10 40 11 40 12 0 print(f"The sum of levels is {sum(gp0['level'])}.") The sum of levels is 325.
Question: I'm trying to install the avatar2 package, but I'm encountering this error: posix_ipc_module.c(37): fatal error C1083: Non è possibile aprire il file inclusione: 'sys/time.h': No such file or directory  I know that it is needed a unix system, bu the framework i'm trying to implement works with angr, which could work also in Windows. Are there any way to install this package on Windows?

Answer: Although not explicitly stated, this package doesn't seem to support Windows as can be seen in the installation instructions provided on their GitHub which uses POSIX style commands. One way you can use this package on Windows is to install Windows Subsystem for Linux (WSL) which provides a compatibility layer that lets you run Linux binary executables natively on Windows. You can build your project using WSL on your Windows machine and install packages in your Linux environment (don't forget to use virtual environments). See How to install Linux on Windows with WSL.
Question: There is a function in NumPy that solves any polynomial with given coefficient (numpy.roots()). So how does NumPy solve it if there is no formula for 5th and higher degree polynomials? I know about Newton's method but I wonder how exactly NumPy applies it. I tried finding information about it in the NumPy documentation and other sources but I did not find anything about that function.

Answer: So, the answer is in numpy documentation. But since that was the opportunity for me to play with it, I put in an answer an experiment illustrating how it can be done. Let's say we want to solve zeros of X**5 - 7*X**4 + 15*X**3 - 5*X**2 - 16*X + 12 I obtained this polynomial by doing import sympy X=sympy.symbols("X") sympy.expand((X-1)*(X-2)**2*(X-3)*(X+1))  So, spoiler alert, we expect to find -1,1,2,3 as zeros, with 2 being a double one. For that, we can build a companion matrix import numpy as np M=np.array([[0,0,0,0,-12], [1,0,0,0,+16], [0,1,0,0,+5], [0,0,1,0,-15], [0,0,0,1,+7]]) #array([[ 0, 0, 0, 0, -12], # [ 1, 0, 0, 0, 16], # [ 0, 1, 0, 0, 5], # [ 0, 0, 1, 0, -15], # [ 0, 0, 0, 1, 7]])  The eigenvalues of this matrix are the zeros of the polynomial (since the characteristic polynomial of this matrix is the one we are interested in). We can check this np.linalg.eigvals(M) #array([-1. , 1. , 1.99999992, 2.00000008, 3. ])  So, that shifted the question: how those eigenvalues are computed? Since, at school, the way we learn to compute eigenvalues is... by solving zeros of characteristic polynomial! So, obviously here that can't be the solution, or else it wouldn't help a lot. There are several method. For example, we could use gram-schmidt to compute both Q and R, such as M=QR def QR(M): Q=np.zeros_like(M) R=np.zeros_like(M) for i in range(len(M)): ei=M[:,i].copy() for j in range(i): R[j,i] = Q[:,j]@M[:,i] ei -= R[j,i]/(Q[:,j]@Q[:,j]) * Q[:,j] R[i,i] = np.sqrt(ei@ei) Q[:,i] = ei / R[i,i] return Q,R  Then the iterative algorithm to compute the eigenvalues A=M.copy() for i in range(100): # That's an arbitrary number of iterations. In real life # we check for convergence. But we aren't in real life: in real life we # we don't code a QR decomposition in pure python :D Q,R=QR(A) A=R@Q np.diag(A) #array([ 3. , 2.02134245, 1.97865755, -1.01636249, 1.01636249])  Result are the eigenvalues of M, that is the zeros of polynomial.
Question: When using all kinds of Python automation tools (such as Playwright and Pyppeteer), I can't seem to grab the "Continue with Google" button on https://dropbox.com/login. When I do this through the console like this: const element = [...document.querySelectorAll('span,button,div,a')].find(el => el.textContent.includes('Doorgaan met Google')); // Check if the element is found and click it if (element) { element.click(); console.log('Button clicked successfully!'); } else { console.error('Button not found!'); }  It finds and clicks the button in Chrome, but not in Firefox. In the programs I've written using Playwright and Pyppeteer, I couldn't find the button either (although sometimes literally just evaluating the same JS code). Anyone know what this is caused by?

Answer: Apparently the structure generated for FireFox is different from the structure generated for other browsers. This worked on FireFox for me: document.querySelector(".L5Fo6c-bF1uUb").click()  In FireFox the button has this structure: <div id="some_id_here" class="L5Fo6c-bF1uUb" tabindex="0"></div>  and no inner text. This is why your script did not find it in FireFox.
Question: With lst = [0] * 10**6 I get times like these:  5.4 ± 0.4 ms bytearray(lst) 5.6 ± 0.4 ms bytes(bytearray(lst)) 13.1 ± 0.7 ms bytes(lst) Python: 3.13.0 (main, Nov 9 2024, 10:04:25) [GCC 14.2.1 20240910] namespace(name='cpython', cache_tag='cpython-313', version=sys.version_info(major=3, minor=13, micro=0, releaselevel='final', serial=0), hexversion=51183856, _multiarch='x86_64-linux-gnu')  I would've expected bytes(lst) and bytearray(lst) to be equally fast or bytearray(lst) to be the slower one, since it's the more complicated type (has more functionality, as it's mutable). But it's the other way around. And even the detour bytes(bytearray(lst)) is much faster than bytes(lst)! Why is bytes(lst) so slow? Benchmark script (Attempt This Online!): from timeit import timeit from statistics import mean, stdev import random import sys setup = 'lst = [0] * 10**6' codes = [ 'bytes(lst)', 'bytearray(lst)', 'bytes(bytearray(lst))' ] times = {c: [] for c in codes} def stats(c): ts = [t * 1e3 for t in sorted(times[c])[:5]] return f'{mean(ts):5.1f} ± {stdev(ts):3.1f} ms ' for _ in range(25): random.shuffle(codes) for c in codes: t = timeit(c, setup, number=10) / 10 times[c].append(t) for c in sorted(codes, key=stats): print(stats(c), c) print('\nPython:') print(sys.version) print(sys.implementation)  Inspired by an answer that also found bytearray to be faster.

Answer: As pointed out by @Homer512 in the comments, bytearray and bytes are implemented quite differently in CPython. While bytearray was given a fast path for creation from list and tuple with GitHub issue #91149, bytes did not receive the same optimization. The said fast path for bytearray takes advantage of the PySequence_Fast_ITEMS, which returns an array of objects, when given a list or a tuple, so that all items can be obtained by simply iterating over the array. The creation of bytes from a list, on the other hand, has to make a PyList_GET_ITEM call to obtain each item in the list with each index, hence the comparatively low performance. I've refactored the code for bytes creation from list and tuple in my PR #128214, which, with a benchmark using: python -m timeit -n 100 -s 'a = [40] * 100000' 'bytes(a)'  shows a ~81% increase in performance: 100 loops, best of 5: 3.43 msec per loop # current 100 loops, best of 5: 1.89 msec per loop # PR #128214  and is indeed faster than the same benchmark constructing bytearray instead: 100 loops, best of 5: 2.14 msec per loop
Question: I have a table my_tbl initialized with datatables2.1.8.js jQuery library: /* using python django template because table template is not in the same html file */ {% include 'my_tbl_template.html' %} let my_tbl=$('#my_tbl').DataTable({...}),  each cell containing <input type="checkbox" value="some_value"> whether initially checked or unchecked; The problem exactly is that my_tbl is not synced with the html dom after changing manually the checkboxes status! (check or uncheck); e.g. when I have two checkboxes (2 cells) initially both checked, when user unchecks one of them printCheckboxStatusDom() prints 1 //(expected number of checked checkboxes) but printCheckboxStatusTbl() prints 2 //the initial checked checkboxes $(document).ready(function(){ $('input[type=checkbox]').change(function() { printCheckboxStatusDom(); // prints correctly how many checkbox is checked at this time printCheckboxStatusTbl(); //always prints initialized values (no change) }); function printCheckboxStatusDom(){ let checkeds = $(document).find('input[type=checkbox]:checked'); console.log('DOM: checked boxes: ' + checkeds.length); } /* Function to print checkboxes status in the my_tbl instance */ function printCheckboxStatusTbl() { my_tbl.rows().every(function (rowIdx, tableLoop, rowLoop) { let rowNode = this.node(); let cellCheckboxes = $(rowNode).find('input[type=checkbox]:checked'); console.log('Tbl: checked boxes: ' + cellCheckboxes.length); } ); } 

Answer: The problem was with DataTables initialization hierarchy in html page! my functions printCheckboxStatusDom(), printCheckboxStatusTbl() and $(document).ready(function(){...}) was stored in a javascript file my_scripts.js in a local dir, and the current page was like  {% include 'my_tbl_template.html' %}; <script>let my_tbl=$('#my_tbl').DataTable({...})</script> <script type="text/javascript" src="my_scripts.js">  so I tried to to change the code to (init datatables at the end):  {% include 'my_tbl_template.html' %} <script type="text/javascript" src="my_scripts.js"> <script>let my_tbl=$('#my_tbl').DataTable({...})</script>  And the problem solved!