eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am currently on macOS 14.3 on Apple silicon. I am trying to install mysqlclient using pip. I already have mysql package installed using brew install mysql pkg-config but my mysqlclient installation is failing due to the errors mentioned below: I am trying to run pip install mysqlclient I am getting the following error:  building 'MySQLdb._mysql' extension creating build/temp.macosx-14.1-arm64-cpython-311 creating build/temp.macosx-14.1-arm64-cpython-311/src creating build/temp.macosx-14.1-arm64-cpython-311/src/MySQLdb clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/opt/mysql-client/include "-Dversion_info=(2, 2, 1, 'final', 0)" -D__version__=2.2.1 -I/Users/avishekde/.pyenv/versions/3.11.6/include/python3.11 -c src/MySQLdb/_mysql.c -o build/temp.macosx-14.1-arm64-cpython-311/src/MySQLdb/_mysql.o -I/opt/homebrew/Cellar/mysql-client/8.3.0/include/mysql -std=c99 src/MySQLdb/_mysql.c:527:9: error: call to undeclared function 'mysql_ssl_set'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] mysql_ssl_set(&(self->connection), key, cert, ca, capath, cipher); ^ src/MySQLdb/_mysql.c:527:9: note: did you mean 'mysql_close'? /opt/homebrew/Cellar/mysql-client/8.3.0/include/mysql/mysql.h:797:14: note: 'mysql_close' declared here void STDCALL mysql_close(MYSQL *sock); ^ src/MySQLdb/_mysql.c:1795:9: error: call to undeclared function 'mysql_kill'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] r = mysql_kill(&(self->connection), pid); ^ src/MySQLdb/_mysql.c:1795:9: note: did you mean 'mysql_ping'? /opt/homebrew/Cellar/mysql-client/8.3.0/include/mysql/mysql.h:525:13: note: 'mysql_ping' declared here int STDCALL mysql_ping(MYSQL *mysql); ^ src/MySQLdb/_mysql.c:2011:9: error: call to undeclared function 'mysql_shutdown'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] r = mysql_shutdown(&(self->connection), SHUTDOWN_DEFAULT); ^ 3 errors generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for mysqlclient Failed to build mysqlclient ERROR: Could not build wheels for mysqlclient, which is required to install pyproject.toml-based projects  The following is the output from pkg-config --cflags --libs mysqlclient -I/opt/homebrew/Cellar/mysql-client/8.3.0/include/mysql -L/opt/homebrew/Cellar/mysql-client/8.3.0/lib -lmysqlclient 

Answer: You can download the source code,and make some modifications, then build & install. Here are the details: https://github.com/PyMySQL/mysqlclient/issues/688
Question: I am working with PyCharm IDE and have encountered an issue where I need to run my Python script within both the PyCharm internal debugger and under proxychains. The reason for using proxychains is that the modules I'm working with do not support working via a SOCKS5 proxy, which is necessary for internet access. Running the script directly from the internal debugger does not provide an option to use proxychains, and using the built-in console does not allow me to step through the code with the debugger. Is there a way to configure PyCharm to run my script with both the internal debugger and proxychains? I want to be able to debug my code while also routing network requests through proxychains. Any help or suggestions on how to achieve this would be greatly appreciated. IDE version: PyCharm 2021.3 (Community Edition) under Linux

Answer: This can be accomplished by creating a shell script which would run your script through ProxyChains and then using this script as an interpreter in PyCharm. First, create a shell script named python in your project directory (let's say it's located at ~/MyProject): #!/bin/sh proxychains4 /usr/bin/python3 "$@"  $@ is used to pass all the arguments (specifically, path to your main script) through this script to the actual Python interpreter (/usr/bin/python3). Make this script executable: chmod +x ~/MyProject/python  Now we can add this script as an interpreter in PyCharm. If you're not using a virtual environment for this project, you can just add a new system interpreter and pick the created python shell script (~/MyProject/python) as the executable. Then add a new configuration using this new interpreter and path to your main script. If you need a virtual environment for this, create a new virtual environment, specify its location and path to your default python interpreter (say, /usr/bin/python3), install all the necessary dependencies inside it and then once again go to Preferences -> Python Interpreter -> Show all, select your newly created virtual environment and change the "interpreter path" field to the shell script path (~/MyProject/python). And one more step. Since PyCharm's debugger uses a built-in local server, you need to exclude connection to it in proxychains.conf: localnet 127.0.0.1/255.255.255.255  Voilà! [proxychains] config file found: /usr/local/etc/proxychains.conf [proxychains] preloading /usr/local/Cellar/proxychains-ng/4.17/lib/libproxychains4.dylib [proxychains] DLL init: proxychains-ng 4.17 [proxychains] DLL init: proxychains-ng 4.17 Connected to pydev debugger (build 233.13763.11) [proxychains] Strict chain ... 171.244.140.160:3991 ... api.ipify.org:443 ... OK {'ip': '171.244.140.160'}
Question: I want to transform my dataframe by merging it cells and summing them into other larger cells given the indices of those, as an example, given the indices [0,2] & [2,4] on the X and Y axis and go from the following dataframe : +----+----+----+----+ | 1 | 2 | 3 | 4 | +----+----+----+----+ | 5 | 6 | 7 | 8 | +----+----+----+----+ | 9 | 10 | 11 | 12 | +----+----+----+----+ | 13 | 14 | 15 | 16 | +----+----+----+----+  to the following one: +----+----+ | 14 | 22 | +----+----+ | 46 | 54 | +----+----+  I was thinking of Pandas' groupBY.transform or rolling would be of help. Any clues?

Answer: Assuming you have homogenous blocks (e.g, 2x2), the most efficient would be to reshape the underlying numpy array and sum: N = 2 out = pd.DataFrame(df.to_numpy() # convert the 2D array to 4D .reshape(len(df)//N, N, -1, N) # sum along dimensions 1 and 3 to go back to 2D .sum((1, 3)) )  If you want non-square blocks (RxC): R, C = 2, 2 out = pd.DataFrame(df.to_numpy() .reshape(len(df)//R, R, df.shape[1]//C, C) .sum((1, 3)) )  Output:  0 1 0 14 22 1 46 54  Intermediate 4D array: # df.to_numpy().reshape((len(df)//N, N, -1, N)) array([[[[ 1, 2], # ──┐ [ 3, 4]], # ─┐├─> 1+2+5+6 = 14 # ││ [[ 5, 6], # ──┘ [ 7, 8]]], # ─┴──> 3+4+7+8 = 22 [[[ 9, 10], # ──┐ [11, 12]], # ─┐├─> 9+10+13+14 = 46 # ││ [[13, 14], # ──┘ [15, 16]]]]) # ─┴──> 11+12+15+16 = 54  shape is not a multiple of N You need to pad the input to a multiple of N. For example using: N = 3 X = int(np.ceil(df.shape[0]/N)*N) Y = int(np.ceil(df.shape[1]/N)*N) df = df.reindex(index=range(X), columns=range(Y), fill_value=0)  Then the aggregation code.
Question: I want to annotate a generic function which takes as arguments another function and its parameters. def forward(func, **kwargs): func(**kwargs)  So, if I have a function which takes two integers: def sum_int(a: int, b: int): ...  in my editor I want help if I pass the wrong object types: forward(sum_int, 1.5, 2.6) # want type checker to complain about using floats instead of integers  How can I annotate forward? Something like: def forward(func: Callable[rest, ret], **kwargs: rest] -> ret: ...  So, the first argument to forward is func and the rest are the keyword arguments. The return is ret. But rest and ret are also the keyword arguments and return type for func. I used to do generics years (decades!) ago in C++, and there were tricks for capturing and unpacking various types, but I don't know whether we have to jump through those kind of hoops with Python, or whether it's even possible. I don't really know what to search for, and didn't turn up anything anywhere near helpful. Thanks!

Answer: You can use typing.ParamSpec to associate the arguments expected by func with the arguments expected by forward, though you do apparently need to use both positional and keyword arguments in the definition of forward: RV = TypeVar('RV') P = ParamSpec('P') def forward(func: Callable[P, RV], *args: P.args, **kwargs: P.kwargs) -> RV: ... def foo(*, a: int, b: int) -> str: ... forward(foo, a=3, b=5) # OK forward(foo, 3, 5) # Not OK, too many positional arguments for foo
Question: What is the best way to ignore NaNs when calculating the mean in Polars? As of polars v0.19.19, the default pl.Expr.mean does not ignore NaNs. Example: test_data = pl.DataFrame( { "group": ["A", "A", "B", "B"], "values": [1, np.nan, 2, 3] } ) test_data.group_by("group").agg(pl.col("values").mean())  results in group values "A" NaN "B" 2.5  Compared to pandas: test_data = pd.DataFrame( { "group": ["A", "A", "B", "B"], "values": [1, np.nan, 2, 3] } ) test_data.groupby("group").mean()  results in group values "A" 1.0 "B" 2.5  The best alternative I can think of is using polars pl.Expr.map_elements and np.nanmean as such: test_data = pl.DataFrame( { "group": ["A", "A", "B", "B"], "values": [1, np.nan, 2, 3] } ) test_data.group_by("group").agg( pl.col("values").map_elements(lambda x: np.nanmean(x.to_numpy())) )  However, the polars API doesn't recommend using it if possible due its lack of speed. Are there faster ways to calculate the mean while ignoring NaNs?

Answer: TLDR. You can simply drop the NaNs before computing the mean. df.group_by("group").agg(pl.col("values").drop_nans().mean())  Output. shape: (2, 2) ┌───────┬────────┐ │ group ┆ values │ │ --- ┆ --- │ │ str ┆ f64 │ ╞═══════╪════════╡ │ A ┆ 1.0 │ │ B ┆ 2.5 │ └───────┴────────┘   Performance comparison against pl.Expr.fill_nan(None). We start by creating an example dataset with 100 million rows and ~20% NaNs in the values column. import random import string N_ROWS = 100_000_000 df = pl.DataFrame({ "group": [random.choice(string.ascii_uppercase) for _ in range(N_ROWS)], "values": [random.random() if random.random() > 0.2 else np.nan for _ in range(N_ROWS)] })  Both methods yield identical results. df_drop = df.group_by("group", maintain_order=True).agg(pl.col("values").drop_nans().mean()) df_fill = df.group_by("group", maintain_order=True).agg(pl.col("values").fill_nan(None).mean()) assert np.allclose(df_drop.select("values").to_numpy(), df_fill.select("values").to_numpy())  Timing both methods yields. %timeit df.group_by("group").agg(pl.col("values").drop_nans().mean()) # 1.21 s ± 28.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  %timeit df.group_by("group").agg(pl.col("values").fill_nan(None).mean()) # 737 ms ± 14.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  Hence, using pl.Expr.fill_nan(None) gives a ~1.6x speed-up. However, unless you are working with dataframes having hundreds of millions of rows, the impact is likely negligible. Note. I've noticed that the relationship above reverses with increasing number of groups. In such cases, efficient parallelisation becomes increasingly important.
Question: I am currently solving an integer programming problem using Pulp. I am aware that the order of statements in Pulp modeling can affect the outcome of the computations. However, I am curious to know if a specific order of modeling can also lead to improvements in computational speed. Is the practice of finding a particular order to enhance calculation speed common? Additionally, I have already implemented optimizations such as eliminating unnecessary variables as much as possible, and dividing the process into pre-processing and post-processing stages. In my case, I noticed a significant improvement in computation time after changing the modeling order: Before changing the order: about 50 seconds After changing the order: about 30 seconds Is there any general advice or guidelines for determining the optimal order of modeling in Pulp to achieve faster computational speeds?

Answer:  Does the Order of Modeling Affect Computational Speed in Pulp for Integer Programming Problems?  Yes. But that's something bad and solvers try hard to decrease this effect. It's also not limited to pulp but applies to discrete-optimization in general. See resource below (focusing on Integer-Programming)!  Is the practice of finding a particular order to enhance calculation speed common?  Absolutely not! If some entities like locations are country-wise sorted, it's absolutely okay (and a good idea) to iterate it the same way. But actively tuning stuff not automatically available... i wouldn't do that. Two remarks:  It's like tuning random-seeds which one should never do In different but similar communities (SAT solvers; similar in terms of "automatic-search"), competitions evaluating different solvers on previously unknown instances randomly permute the model rows afaik!   Before changing the order: about 50 seconds After changing the order: about 30 seconds  This sounds like a statistical-evaluation with a sample-size of 1. This does not tell much. It probably won't even survive a re-run with a different seed in the solver.  If your problem is as sensitive as you claim, it's usually indicating, that your model is not good enough. It basically means, that your solver is very much depending on luck! Other formulations (not the order) might be better but this is always problem-dependent. If there are some insights which can help the solver, there are also more robust methods of giving hints (branch-heuristics and co.) although i'm not sure how much pulp supports here.  For some background:  Performance variability in mixed integer programming (PDF)  e.g. page 9: "First variability generator: permutations"
Question: I've graphed my original function and I want to graph the tangent line at the point of interest. import numpy as np from sympy import lambdify, symbols, diff, Abs point_of_interest = 8 graphRange = [1,15] # define the variable and the function xsym = symbols('x') # With a chord length of 60, plot the circle diameters origFunction = 2 * ((60 ** 2) / (8 * Abs(xsym)) + Abs(xsym) / 2) # define the derivative derivative = diff(origFunction, xsym) # define the tangent line at point of interest tangentLine = derivative.subs(xsym, point_of_interest) * (xsym - point_of_interest) + origFunction.subs(xsym, point_of_interest) # Convert the SymPy function to a lambda function origF = lambdify(xsym, origFunction, "numpy") # Generate x values x_values = np.linspace(graphRange[0], graphRange[1], 100) # Generate y-values for the original function y_values = origF(x_values) # Plot the original function plt.plot(x_values, y_values, label='Original Function') # THIS SECTION DOESN'T WORK YET # Convert the SymPy function to a lambda function diffF = lambdify(xsym, tangentLine, "numpy") # Generate y-values for the tangent line y_tang_values = diffF(x_values) # Plot the tangent line plt.plot(x_values, y_tang_values, label='Tangent Line') #plot the point of interest plt.plot(point_of_interest, origF(point_of_interest), 'ro', label='Point of Interest') # Add labels and legend plt.xlabel('x') plt.ylabel('y') plt.title('Graph of Original Function and Tangent Line') plt.legend() # Show the plot plt.show()  The error I get is this Traceback (most recent call last): File "x:\Python Projects\Chord calc.py", line 37, in y_tang_values = diffF(x_values) ^^^^^^^^^^^^^^^ File "", line 4, in _lambdifygenerated NameError: name 'Subs' is not defined I don't know what to do to fix this error.

Answer: At the moment, you are using a complex symbol, xsym: it doesn't have any assumptions, hence it is as general as it gets. Your function contains Abs(xsym): because xsym is complex, its derivative will be quite convoluted: print(Abs(xsym).diff(xsym)) # (re(x)*Derivative(re(x), x) + im(x)*Derivative(im(x), x))*sign(x)/x  Once you substitutes a numeric value into that expression, Subs appears, and you get the error later on. The solution is simple: create a real symbol. Replace this line: xsym = symbols('x')  with: xsym = symbols('x', real=True)  Then, everything will work as expected.
Question: For some quick tests (this code will evolve later anyway, so a temporary solution is ok), I need to use Matplotlib in a thread, and not the main thread. Usually we have this warning but it works anyway:  UserWarning: Starting a Matplotlib GUI outside of the main thread will likely fail.  Here it is indeed the case. However, when the first plot is closed, and we do the same a second time, then it totally fails with not a warning, but an error:  RuntimeError: main thread is not in main loop  Is there a way to make the following code work? (even if it is not the recommened matplotlib way) import numpy as np import matplotlib.pyplot as plt import threading, time def visualization_thread(): fig = plt.figure() ax = fig.add_subplot(111) l1, *_ = ax.plot(y1, color='r', label="1") fig.show() fig.canvas.flush_events() while running: l1.set_ydata(y1) fig.canvas.draw_idle() fig.canvas.flush_events() plt.pause(0.020) def data_thread(): global y1, y2, y3 while running: y1 = np.random.rand(100) time.sleep(0.020) running = True # !!! WARNING but it still works threading.Thread(target=data_thread).start() threading.Thread(target=visualization_thread).start() time.sleep(4) running = False time.sleep(2) running = True # !!! FAILS, why? threading.Thread(target=data_thread).start() threading.Thread(target=visualization_thread).start() 

Answer: I just noticed that adding plt.close() at the end of the visualization thread solves the problem. Again, this is a temporary solution, since matplotlib is not threadsafe anyway.
Question: Please review this code where I have used CustomTkinter (alias as "ctk") to create a button. The "command" field contains a function "selectfile". I want to implement this "selectfile" function. button_to_select = ctk.CTkButton(app, text = "Choose file", fg_color = "blue", command = selectfile) button_to_select.pack(padx = 25, pady = 25) button_to_select.place(x = 825, y = 300)  So we have always seen applications opening up another window for other works such as Logins or File Selections. I want to create the same here. The button on click will open up a window of the Windows Explorer and let users select any number and any type of files. We will also have an "Ok" button that will close the window and send the files back to the application for further work. How to implement this function? So it will be like when we want to send some files into WhatsApp, the Windows Explorer opens and we select the files... This would be something similar. And no is not an Web-app. It is a Native App. I am unable to create anything because I lack knowledge. All I tried to do is use the "subprocess" module to run the Windows Explorer. But that just runs the explorer separately and does not do the expected work as I described.

Answer: You can use filedialog. Use this function to open the windows file selector from customtkinter import filedialog def selectfile(): filename = filedialog.askopenfilename() print(filename)  Edit : Customtkinter got this function
Question: I am running a script that uses pd.to_datetime() on inputs that are sometime not able to be parsed. For example if I try to run pd.to_datetime('yesterday') it results to an error DateParseError: Unknown datetime string format, unable to parse: yesterday, at position 0 I 'd like to catch this exception and process it further in my code. I have tried: try: pd.to_datetime('yesterday') except pd.errors.ParserError: print('exception caught')  but the exception is not caught. Does anyone know where DateParseError is defined and how I can catch it? Searching in pandas documentation doesn't yield any results

Answer: You can import it from pandas._libs.tslibs.parsing: from pandas._libs.tslibs.parsing import DateParseError try: pd.to_datetime('yesterday') except DateParseError: print('exception caught')
Question: I have a dataframe: df = pl.DataFrame( { "t_left": [0.0, 1.0, 2.0, 3.0], "t_right": [1.0, 2.0, 3.0, 4.0], "counts": [1, 2, 3, 4], } )  And I want to map each row into a list, and then collect all values (to be passed into e.g. matplotlib.hist) I can do it by hand like: times = [] for t_left, t_right, counts in df.rows(): times.extend(np.linspace(t_left, t_right, counts + 1)[1:])  but this is painfully slow for my data set. I am completely new to both python and polars, so I was wondering if there is a better way to achieve this. EDIT Complete copy-paste example to reproduce. import polars as pl import numpy as np size = 1000000 df = pl.DataFrame( { "t_left": np.random.rand(size), "t_right": np.random.rand(size) + 1, "counts": [1] * size, } ) times = [] for t_left, t_right, counts in df.rows(): times.extend(np.linspace(t_left, t_right, counts + 1)[1:]) 

Answer: As you need the final result as one big list, you don't really need to work with lists, you can just explode() your dataframe into multiple rows so then you can run vectorized operations on it: so, taking your example data: df = pl.DataFrame( { "t_left": [0.0, 1.0, 2.0, 3.0], "t_right": [1.0, 2.0, 3.0, 4.0], "counts": [1, 2, 3, 4], } ) ┌────────┬─────────┬────────┐ │ t_left ┆ t_right ┆ counts │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ i64 │ ╞════════╪═════════╪════════╡ │ 0.0 ┆ 1.0 ┆ 1 │ │ 1.0 ┆ 2.0 ┆ 2 │ │ 2.0 ┆ 3.0 ┆ 3 │ │ 3.0 ┆ 4.0 ┆ 4 │ └────────┴─────────┴────────┘  Now, let's create some columns we can use for our calculation. To calculate evenly distributed numbers you need a starting point - t_left, step - (t_right - t_left) / counts and then you need to repeat it counts times. # Thanks @Hericks for pointing out that code could be simplified even more times = df.select( start = pl.col('t_left'), step = ((pl.col("t_right") - pl.col("t_left")) / pl.col("counts")), i = pl.int_ranges(1, pl.col('counts') + 1) ).explode('i') # my old code # times = df.select( # start = pl.col('t_left').repeat_by('counts').explode(), # step = ((pl.col("t_right") - pl.col("t_left")) / pl.col("counts")).repeat_by('counts').explode(), # i = pl.int_ranges(1, pl.col('counts') + 1).explode() #) ┌───────┬──────────┬─────┐ │ start ┆ step ┆ i │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ i64 │ ╞═══════╪══════════╪═════╡ │ 0.0 ┆ 1.0 ┆ 1 │ │ 1.0 ┆ 0.5 ┆ 1 │ │ 1.0 ┆ 0.5 ┆ 2 │ │ 2.0 ┆ 0.333333 ┆ 1 │ │ … ┆ … ┆ … │ │ 3.0 ┆ 0.25 ┆ 1 │ │ 3.0 ┆ 0.25 ┆ 2 │ │ 3.0 ┆ 0.25 ┆ 3 │ │ 3.0 ┆ 0.25 ┆ 4 │ └───────┴──────────┴─────┘  now, it's just a matter of applying the formula and converting result to_list(): times.select( res = pl.col('start') + pl.col('step') * pl.col('i') )['res'].to_list() [1.0, 1.5, 2.0, 2.3333333333333335, 2.6666666666666665, 3.0, 3.25, 3.5, 3.75, 4.0]
Question: I am working with a Polars DataFrame that contains a column with values like this: shape: (3, 2) ┌──────────────┬──────────────┐ │ A │ B │ │ --- │ --- │ │ str │ str │ ├──────────────┼──────────────┤ │ A|B:0d:cs │ C:2ew2 │ ├──────────────┼──────────────┤ │ A:1ds0 │ E|F:91we23 │ ├──────────────┼──────────────┤ │ QW|P:3dwsd │ A|Z:12w219 │ └──────────────┴──────────────┘  I want to extract the left substring from each cell in the DataFrame until I find the character ":". The desired result should look like this: shape: (3, 2) ┌──────────────┬──────────────┐ │ A │ B │ │ --- │ --- │ │ str │ str │ ├──────────────┼──────────────┤ │ A|B │ C │ ├──────────────┼──────────────┤ │ A │ E|F │ ├──────────────┼──────────────┤ │ QW|P │ A|Z │ └──────────────┴──────────────┘  Thank you for your help in advance!

Answer: you can use str.split() and list.first(): df.select(pl.all().str.split(':').list.first()) ┌─────────┬───────┐ │ integer ┆ float │ │ --- ┆ --- │ │ str ┆ str │ ╞═════════╪═══════╡ │ A|B ┆ C │ │ A ┆ E|F │ │ QW|P ┆ A|Z │ └─────────┴───────┘
Question: I would like to disable/suppress the ruff linter (or certain linting rules) on a block of code. I know that I can do this for single lines (by using # noqa: <rule_code> at the end of the line) or for entire files/folder (#ruff: noqa <rule_code> at the top of the file). However, I would like to disable the linter ruff on one function or on a multi-line block of code, but not an entire file. Is there a way to do this, without adding noqa to the end of each line?

Answer: Not currently possible (as of 29 Jan 2024). See here for updates on the issue