eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to compute some derivatives of neural network outputs. To be precise I need the jacobian matrix of the function that is represented by the neural network and the second derivative of the function with respect to its inputs. I want to multiply the derivative of the jacobian with a vector of same size as the input for every sample. I got the result tht I need with this implementation: import torch x_1 = torch.tensor([[1.,1.,1.]], requires_grad=True) x_2 = torch.tensor([[2.,2.,2.]], requires_grad=True) # Input to the network with dim 2x3 --> 2 Samples 3 Feature x = torch.cat((x_1,x_2),dim=0) def calculation(x): c = torch.tensor([[1,2,3],[4,5,6],[7,8,9]]).float() return (x@c)**2 c = torch.tensor([[1,2,3],[4,5,6],[7,8,9]]).float() #output of the network with dimension 2x3 --> 3 outputs per Sample y = calculation(x) #Calculation of my jacobian with dimension 2x3x3 (one for each sample) g = torch.autograd.functional.jacobian(calculation,(x),create_graph=True) jacobian_summarized = torch.sum(g, dim=0).transpose(0,1) #Calculation of my second order derivative 2x3x3x3 (On Tensor for each Sample) gg = torch.autograd.functional.jacobian(lambda x: torch.sum(torch.autograd.functional.jacobian(calculation, (x), create_graph=True), dim=0).transpose(0,1), (x), create_graph=True) second_order_derivative = torch.sum(gg, dim=0).transpose(1,2).transpose(0,1) print('x:', x) print('c:',c) print('y:',y) print('First Order Derivative:',jacobian_summarized) print('Second Order Derivative:',second_order_derivative) # Multiplication with for loop result = torch.empty(0) for i in range(y.shape[0]): result_row = torch.empty(0) for ii in range(y.shape[1]): result_value = (x[i].unsqueeze(0))@second_order_derivative[i][ii]@(x[i].unsqueeze(0).T) result_row = torch.cat((result_row, result_value), dim=1) result = torch.cat((result, result_row)) print(result)  I would like to know if there is a way to get the same result of the multiplication without having to use 2 for loops but rather some simple multiplication of the matrices

Answer: It seems like you're looking for einsum. Should be something like: result = torch.einsum('bi,bijk,bk->bj', x, second_order_derivative, x)
Question: I am trying to write a method that evaluates a statement, but the operator (>, <, =) is sent by the user. I am wondering if there is a easy way to write a more concise method. The simplified version of the code is: def comparsion(val1: int, val2: int, operator: str): if operator == ">": if val1 > val2: return True elif operator == "=": if val1 == val2: return True elif operator == "<": if val1 < val2: return True  The only thing that changes in each case is the operator, I am wondering if there is a more efficient way to write this.

Answer: Use the operator module, which provides named functions for all the standard operators. Then you can use a dictionary to map from the operator string to the corresponding function. from operator import lt, gt, eq ops = {'<': lt, '>': gt, '=': eq} def comparsion(val1: int, val2: int, operator: str): return ops[operator](val1, val2)
Question: I am trying to read a large number of zipped files (.zip or .docx) in a loop, each again containing a large number of embedded XML (.xml) files inside them. However some of the embedded XML files are broken/corrupted. I can create a parser which ignores the errors and loads the XML contents. However, I want to know which XML file is corrupted and which particular element inside it is broken/failing. I have tried the below code: import re import os import zipfile from lxml import etree for file in os.listdir(filepath): if file.endswith('.zip') or file.endswith('.docx'): ext = os.path.splitext(file)[1] newfile = f"{os.path.splitext(os.path.basename(file))[0]}_new{ext}" zippedin = zipfile.ZipFile(os.path.join(filepath, file), 'r') recovering_parser = etree.XMLParser(recover=True) matched_items = [] for item in zippedin.infolist(): xmltree = etree.fromstring(zippedin.read(item.filename), parser=recovering_parser) for node in xmltree.iter(tag=etree.Element): if re.search('XXXXXXX', str(node)) or re.search('YYYYYYYY', str(node.attrib)): matched_items.append(item) with zipfile.ZipFile(os.path.join(filepath, newfile), 'w') as zippedout: for element in matched_items: zippedout.writestr(element, zippedin.read(element.filename)) zippedin.close()  This code snippet works perfectly fine and bypasses the broken XML files inside the zipped archives. However, I require to identify which files are failing and also the individual components. If I remove the recovering_parser portion, I receive error messages of the following sort: lxml.etree.XMLSyntaxError: xmlns: 'ABCDEFGHXXXX' is not a valid URI, line 5, column 45  It does not show which XML is corrupted. Can someone help me identify the broken XMLs and a proper way of exception handling and error scraping/extracting the faulty component name.

Answer: Using recovering_parser = etree.XMLParser(recover=True) is preventing you from being able to catch which files are broken. In order to catch those errors, you can use a try/except block. import re import os import zipfile from lxml import etree try: # xml parsing code here except Exception as e: # Debugging code here print(file) # print the file name
Question: Not a duplicate of Zero pad numpy array (that I posted 9 years ago, ouch!) because here it's about n-dimensional arrays. How to zero pad a numpy n-dimensional array, if possible in one line? Example: a = np.array([1, 2, 3]) zeropad(a, 8) # [1, 2, 3, 0, 0, 0, 0, 0] b = np.array([[1, 2], [3, 4], [5, 6]]) zeropad(b, (5, 2)) # [[1, 2], [3, 4], [5, 6], [0, 0], [0, 0]]  When using b.resize((5, 2)), here it works, but in some real cases, it gives: ValueError: cannot resize this array: it does not own its data  How to zero pad numpy nd arrays no matter if it owns its data or not?

Answer: Instead of using pad, since you want to pad after, you could create an array of zeros and assign the existing values: out = np.zeros(pad, dtype=arr.dtype) out[np.indices(arr.shape, sparse=True)] = arr  Or, if you only want to pad the first dimension, with resize. Just ensure that the array owns its data with copy: out = arr.copy() out.resize(pad)  Output: array([[1, 2], [3, 4], [5, 6], [0, 0], [0, 0]])  I really want a one-liner resize: IMO there is no good reason for that, but you could always use an assignment expression (python ≥ 3.8): (out:=arr.copy()).resize(pad)  Output for a different pad arr = np.array([[1, 2], [3, 4], [5, 6]]) pad = (5, 3) # output zeros + assignment array([[1, 2, 0], [3, 4, 0], [5, 6, 0], [0, 0, 0], [0, 0, 0]]) # output resize array([[1, 2, 3], [4, 5, 6], [0, 0, 0], [0, 0, 0], [0, 0, 0]])
Question: Using a data frame of dates and values starting from 1 Jan 2022: import datetime as dt import altair as alt import polars as pl import numpy as np alt.renderers.enable("browser") dates = pl.date_range(dt.date(2022, 1, 1), dt.date(2025, 1, 22), "1d", eager = True) values = np.random.uniform(size = len(dates)) df = pl.DataFrame({"dates": dates, "values": values}) alt.Chart(df).mark_point().encode(alt.X("dates:T"), alt.Y("values:Q")).show()   But if I start the data frame from 2020 and filter it for dates > 1 Jan 2022: dates_b = pl.date_range(dt.date(2020, 1, 1), dt.date(2025, 1, 22), "1d", eager = True) values_b = np.random.uniform(size = len(dates_b)) df_b = pl.DataFrame({"dates": dates, "values": values}) alt.Chart(df_b.filter(pl.col("dates") > dt.date(2022, 1, 1))).mark_point().encode(alt.X("dates:T"), alt.Y("values:Q")).show()   How can I specify that years must be shown? Note that I do get the right result if I filter using >= to include 1 Jan 2022, but that's besides the point. I always need years.

Answer: You can use labelExpr to build your own logic for setting tick labels. For example, this gives the year if the month is January and the month otherwise. dates_b = pl.date_range(dt.date(2020, 1, 1), dt.date(2025, 1, 22), "1d", eager=True) values_b = np.random.uniform(size=len(dates_b)) df_b = pl.DataFrame({"dates": dates, "values": values}) alt.Chart(df_b.filter(pl.col("dates") > dt.date(2022, 1, 1))).mark_point().encode( alt.X("dates:T").axis( labelExpr="timeFormat(datum.value, '%m') == '01' ? timeFormat(datum.value, '%Y') : timeFormat(datum.value, '%b')", ), alt.Y("values:Q"), )
Question: I have a long build script which is a mix of "real" python code and lengthy subprocess.run() calls to things like debootstrap, wget, apt-get or build.sh. Everything is parallelized. One thread does debootstrap followed by apt-get, while another does build.sh, then both are joined and we start more threads for combining the results and so on. Each thread logs to a dedicated file. While looking for a generic way to log the output of subprocess.run() started by one of these threads, I came upon the following answer: https://stackoverflow.com/a/31868783/876832 Is it a bug to call subprocess.run() in a python script with more than one thread?

Answer:  Is it a bug to call subprocess.run() in a python script with more than one thread?  No. I do that all the time. I think the answer you linked is misguided. After all, you don't even know whether subprocess.Popen() and its facades like subprocess.run() use the fork syscall (especially on Windows, they certainly don't, since that call doesn't exist there). Sure, if you were to manually os.fork(), I'd exercise caution in a threaded program (and indeed, as the docs say, since 3.12, if Python detects you're doing that, it'll warn you). Under the hood, though, on POSIX systems, subprocess.Popen() uses os.posix_spawn when possible, and otherwise a very careful fork_exec implementation. Oh, and as an aside, the question you linked actually uses the Python 2.7-era low-level thread module, these days aptly called _thread, the low-level threading API. That's a different low-level beast than threading.Threads.
Question: I need some help with python regex. My "payload" string look like this "status=0/eStopStatus=2/version=1.0.16/runTime=005320" My code uses the following parse out a list of variables and values: variables = re.findall(r'([\w]+)=', payload) values = re.findall(r"[-+]?(?:\d*\.*\d+)", payload)  My variables are parsed correctly as: status eStopStatus version runTime  But the double . in my version value 1.0.16 trips up my values and I get 0 2 1.0 .16 005320  I must admit I am battling to understand the syntax of the parameters using regex for this. Any assistance appreciated

Answer: This will work perfectly: values = re.findall(r"[-+]?(\d+(?:\.\d+)*)", payload)  Here is a link to the pattern and test string: https://regex101.com/r/YbcMUR/1 TEST CODE: import re payload = "status=0/eStopStatus=2/version=1.0.16/runTime=005320" keywords_pattern = r'([\w]+)=' values_pattern = r'[-+]?(\d+(?:\.\d+)*)' variables = re.findall(keywords_pattern, payload) values = re.findall(values_pattern, payload) [print(variable) for variable in variables] [print(value) for value in values]  RESULTS: PS C:\..\regex_payload.py status eStopStatus version runTime 0 2 1.0.16 005320
Question: I have two datafames: df1 = pd.DataFrame({ 'from': [0, 2, 8, 26, 35, 46], 'to': [2, 8, 26, 35, 46, 48], 'int': [2, 6, 18, 9, 11, 2]}) df2 = pd.DataFrame({ 'from': [0, 2, 8, 17, 34], 'to': [2, 8, 17, 34, 49], 'int': [2, 6, 9, 17, 15]})  I want to create a new dataframe that looks like this: df = pd.DataFrame({ 'from': [0, 2, 8, 17, 26, 34, 35, 46, 48], 'to': [2, 8, 17, 26, 34, 35, 46, 48, 49], 'int': [2, 6, 9, 9, 8, 1, 11, 2, 1]})  I have tried standard merging scripts but have not been able to split the rows containing higher 'from' and 'to' numbers in either df1 or df2 into smaller ones. How can I merge my dataframes over multiple columns and split rows?

Answer: Frirst, combine all unique from and to values from both df1 and df2 to create a set of breakpoints: breakpoints = set(df1['from']).union(df1['to']).union(df2['from']).union(df2['to']) breakpoints = sorted(breakpoints)  In the example, this is [0, 2, 8, 17, 26, 34, 35, 46, 48, 49]. Now, create a new dataframe with these from and to values, then compute the intervals: new_df = pd.DataFrame({'from': breakpoints[:-1], 'to': breakpoints[1:]}) new_df['int'] = new_df['to'] - new_df['from']  Result:  from to int 0 0 2 2 1 2 8 6 2 8 17 9 3 17 26 9 4 26 34 8 5 34 35 1 6 35 46 11 7 46 48 2 8 48 49 1
Question: The mod operator in Free Pascal does not produce the results I would expect. This can be demonstrated by the program below whose output does not agree with the result of the same calculation in Python (or Google). program test(output); var a, b, c: longint; begin a := -1282397916; b := 2147483647; c := a mod b; writeln (c:16); end.   -1282397916  Compare this to the output of the Python script below which gives the result I expected. #!/usr/bin/python a = -1282397916 b = 2147483647 c = a % b print (c)  865085731  This is the same as the result as that obtained by pasting the following text into Google (and when using the mod operator in VAX‑Pascal). (-1282397916 % 2147483647)  The question is: Why does Free Pascal behave differently? And how do I get the same result as that obtained when using the mod operator in Python?

Answer: With respect to a non‑negative integral dividend and positive integral divisor there is no ambiguity. All programming languages do the same. Once you use other values though, programming languages differ. Pascal uses a Euclidean‑like definition of the modulus operator. In ISO standard 7185 (“Standard Pascal”), page 48, it is defined as follows:  A term of the form i mod j shall be an error if j is zero or negative; otherwise, the value of i mod j shall be that value of (i − (k × j)) for integral k such that 0 ≤ i mod j < j.  In other words:  Evaluation of a term of the form x mod y is an error if y is less than or equal to zero; otherwise there is an integer k such that x mod y satisfies the following relation:  0 <= x mod y = x − k * y < y.   Source: Jensen, Kathleen; Wirth, Niklaus. Pascal – user manual and report (4th revised ed.). p. 168. doi:10.1007/978‑1‑4612‑4450‑9. ISBN 978‑0‑387‑97649‑5. Thus the result of the mod operator is guaranteed to be non-negative. Unfortunately, as you have already observed, the FreePascal Compiler does not adhere to the ISO standards. The FPC will only return the proper result if {$modeSwitch isoMod+} is set: program moduloConfusion(output); {$modeSwitch isoMod+} type integer = ALUSInt; var dividend, divisor: integer; begin dividend := -1282397916; divisor := 2147483647; writeLn(dividend mod divisor:16) end.  Note, this affects the definition of the mod operator per compilation unit, so the RTL and everything else – unless recompiled – continues using the other definition internally. Rest assured, however, Delphi and the GPC (GNU Pascal Compiler) do work correctly without making jump through loops.  If you want to get the same result as in Python, you need to define and use your own function (here in Extended Pascal): function modulo(protected dividend, divisor: integer): integer; begin modulo := (abs(dividend) mod abs(divisor)) * -1 pow ord(divisor < 0) end;  There is no magic switch to make FreePascal’s mod behave exactly like Python’s %.
Question: My question is that are you able to run any code after the if __name__ == "__main__": app.run()  in flask. I tried to print something after the above syntax to check and it only appeared after I exited the process. I was unable to find answer to this question. If yes, please guide me on how to accomplish this. I need to run a block of code after the website has been launched. I am running on localhost if that helps. PS: I am creating a website that downloads an image from the user's computer, runs a python script and predicts what the thing in the image is and tells whether the object is recyclable or not. This is for a school project. I tried to print("check")  and, as I use pycharm, stopped the process using the stop button and the following appeared:- WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on http://127.0.0.1:5000 Press CTRL+C to quit check Process finished with exit code 0 

Answer: As said by @JonSG and @Chris, I needed to create a route method before the app.run()  to make the code work. Thanks!
Question: Input (New York Time in string format) = '2024-11-01 13:00:00' Output (UTC Time in string format) = '2024-11-01 17:00:00'

Answer: Parse the time. It won't be "time zone aware", so apply local time zone, convert to UTC and format again: import datetime as dt import zoneinfo as zi zone = zi.ZoneInfo('America/New_York') fmt = '%Y-%m-%d %H:%M:%S' s = '2024-11-01 13:00:00' print(s) # parse the time and apply the local time zone nyt = dt.datetime.strptime(s, fmt).replace(tzinfo=zone) # convert to UTC and format string utc = nyt.astimezone(dt.UTC).strftime(fmt) print(utc)  Output: 2024-11-01 13:00:00 2024-11-01 17:00:00
Question: I just realized I had a typo in my membership test and was worried this bug had been causing issues for a while. However, the code had behaved just as expected. Example: "test" in "testing" in "testing" in "testing"  This left me wondering how this membership expression works and why it's allowed. I tried applying some order of operations logic to it with parentheses but that just breaks the expression. And the docs don't mention anything about chaining. Is there a practical use case for this I am just not aware of?

Answer: in is a comparison operator. As described at the top of the section in the docs you linked to, all comparison operators can be chained:  Formally, if a, b, c, …, y, z are expressions and op1, op2, …, opN are comparison operators, then a op1 b op2 c ... y opN z is equivalent to a op1 b and b op2 c and ... y opN z, except that each expression is evaluated at most once.  So: "test" in "testing" in "testing" in "testing"  Is equivalent to: "test" in "testing" and "testing" in "testing" and "testing" in "testing"
Question: I have a regex on my input parameter: r"^(ABC-\d{2,9})|(ABz?-\d{3})$"  Ideally it should not allow parameters with ++ or -- at the end, but it does. Why is the regex not working in this case but works in all other scenarios? ABC-12 is a valid. ABC-123456789 is a valid. AB-123 is a valid. ABz-123 is a valid. 

Answer: The problem is that your ^ and $ anchors don't apply to the entire pattern. You match ^ only in the first alternative, and $ only in the second alternative. So if the input matches (ABC-\d{2,9}) at the beginning, the match will succeed even if there's more after this. You can put a non-capturing group around everything except the anchors to fix this. r"^(?:(ABC-\d{2,9})|(ABz?-\d{3}))$"
Question: I have a pandas dataframe: import pandas as pd import numpy as np np.random.seed(150) df = pd.DataFrame(np.random.randint(0, 10, size=(10, 2)), columns=['A', 'B'])  I want to add a new column "C" whose values ​​are the combined-list of every three rows in column "B". So I use the following method to achieve my needs, but this method is slow when the data is large. >>> df['C'] = [df['B'].iloc[i-2:i+1].tolist() if i >= 2 else None for i in range(len(df))] >>> df A B C 0 4 9 None 1 0 2 None 2 4 5 [9, 2, 5] 3 7 9 [2, 5, 9] 4 8 3 [5, 9, 3] 5 8 1 [9, 3, 1] 6 1 4 [3, 1, 4] 7 4 1 [1, 4, 1] 8 1 9 [4, 1, 9] 9 3 7 [1, 9, 7]  When I try to use the df.apply function, I get an error message: df['C'] = df['B'].rolling(window=3).apply(lambda x: list(x), raw=False) TypeError: must be real number, not list  I remember that pandas apply doesn't seem to return a list, so is there a better way? I searched the forum, but couldn't find a suitable topic about apply and return.

Answer: You can use numpy's sliding_window_view: from numpy.lib.stride_tricks import sliding_window_view as swv N = 3 df['C'] = pd.Series(swv(df['B'], N).tolist(), index=df.index[N-1:])  Output:  A B C 0 4 9 NaN 1 0 2 NaN 2 4 5 [9, 2, 5] 3 7 9 [2, 5, 9] 4 8 3 [5, 9, 3] 5 8 1 [9, 3, 1] 6 1 4 [3, 1, 4] 7 4 1 [1, 4, 1] 8 1 9 [4, 1, 9] 9 3 7 [1, 9, 7]
Question: In the configuration management library Hydra, it is possible to only partially instantiate classes defined in configuration using the _partial_ keyword. The library explains that this results in a functools.partial. I wonder how this interacts with seeding. E.g. with  pytorch torch.manual_seed() lightnings seed_everything etc.  My reasoning is, that if I use the _partial_ keyword while specifying all parameters for __init__, then I would essentially obtain a factory which could be called after specifying the seed to do multiple runs. But this assumes that _partial_ does not bake the seed in already. To my understanding that should not be the case. Is that correct?

Answer: Before using hydra.utils.instantiate no third party code is not run by hydra. So you can set your seeds before each use of instantiate; or if a partial before each call to the partial. Here a complete toy example, based on Hydra's doc overview, which creates a partial to instantiate an optimizer or a model, that takes a callable optim_partial as an argument. # config.yaml model: _target_: "__main.__.MyModel" optim_partial: _partial_: true _target_: __main__.MyOptimizer algo: SGD lr: 0.01  from functools import partial from typing import Callable import random from pprint import pprint import hydra from omegaconf import DictConfig, OmegaConf class MyModel: def __init__(self, lr, optim_partial: Callable[..., "MyOptimizer"]): self.optim_partial = optim_partial self.optim1 = self.optim_partial() self.optim2 = self.optim_partial() class MyOptimizer: def __init__(self, algo): print(algo, random.randint(0, 10000)) @hydra.main(config_name="config", config_path="./", version_base=None) def main(cfg: DictConfig): # Check out the config pprint(OmegaConf.to_container(cfg, resolve=False)) print(type(cfg.model.optim_partial)) # Create the functools.partial optim_partial: partial[MyOptimizer] = hydra.utils.instantiate(cfg.model.optim_partial) # Set the seed before you call the a partial random.seed(42) optimizer1: MyOptimizer = optim_partial() optimizer2: MyOptimizer = optim_partial() random.seed(42) optimizer1b: MyOptimizer = optim_partial() optimizer2b: MyOptimizer = optim_partial() # model is not a partial; use seed before creation random.seed(42) model: MyModel = hydra.utils.instantiate(cfg.model) if __name__ == "__main__": main()  # Output {'model': {'_target_': '__main__.MyModel', 'lr': 0.01, 'optim_partial': {'_partial_': True, '_target_': '__main__.MyOptimizer', 'algo': 'SGD'}}} type of cfg.model.optim_partial <class 'omegaconf.dictconfig.DictConfig'> SGD 1824 SGD 409 SGD 1824 SGD 409 SGD 1824 SGD 409
Question: tensorflow python package installed using poetry is not recognised within python. (poetry version - 2.0.1) Anyone else facing this issue? and how did you solve them? C:\Users\username\Documents\folder>poetry add tensorflow  Using version ^2.18.0 for tensorflow Updating dependencies Resolving dependencies... (0.9s) Package operations: 18 installs, 0 updates, 0 removals  Installing markupsafe (3.0.2) Installing grpcio (1.70.0) Installing markdown (3.7) Installing protobuf (5.29.3) Installing setuptools (75.8.0) Installing tensorboard-data-server (0.7.2) Installing werkzeug (3.1.3) Installing wheel (0.45.1) Installing astunparse (1.6.3) Installing gast (0.6.0) Installing libclang (18.1.1) Installing opt-einsum (3.4.0) Installing google-pasta (0.2.0) Installing flatbuffers (25.1.24) Installing termcolor (2.5.0) Installing tensorboard (2.18.0) Installing wrapt (1.17.2) Installing tensorflow (2.18.0)  Writing lock file C:\Users\username\Documents\folder>poetry run python Python 3.12.8 (tags/v3.12.8:2dc476b, Dec 3 2024, 19:30:04) [MSC v.1942 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import tensorflow Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'tensorflow' 

Answer: tensorflow-intel has not been installed, which since you are on windows it should be. Please go to https://github.com/tensorflow/tensorflow/issues/75415 and encourage the tensorflow folk to put consistent metadata in all of their wheels so that cross-platform resolvers like poetry and uv can reliably derive accurate information.