eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying custom model managers to add annotations to querysets. My problem, which started as a little annoyance but I now realize can be an actual problem, is that VS Code does not recognise the methods defined in the custom model manager/queryset. Example: from django.db import models from rest_framework.generics import ListAPIView # models.py class CarQuerySet(models.QuerySet): def wiht_wheels(self): # NOTE: intentional typo pass # assume this does some annotaion class Car(models.Model): objects = CarQuerySet.as_manager() # views.py class ListCarsView(ListAPIView): def get_queryset(self): return Car.objects.wiht_weels() # <--- white instead of yellow  At first, I was just annoyed by the fact that wiht_weels is printed in white as opposed to the usual yellow for methods/functions. Then I was more annoyed because this means VS Code will not give me any hints as to what args the method expects or what it returns. Finally, I accidentally made a typo on a name of one of these custom model methods, I hit refactor->rename, but it only renamed it in place, not on the places where it is used (views), probably because VS Code doesn't understand that method is being used anywhere. Is there a solution to this?

Answer: Add a few type annotations to your class methods:  from django.db import models from rest_framework.generics import ListAPIView # models.py class CarQuerySet(models.QuerySet): def with_wheels(self) -> "CarQuerySet": # quotes here pass class CarManager(models.Manager): def get_queryset(self) -> CarQuerySet: # update the definition here return CarQueryset(self.model, using=self._db) def with_wheels(self) -> CarQuerySet: # redefine method here return self.with_wheels(self): class Car(models.Model): objects: CarManager = CarManager # add a type hint here  vs code will now show you your customer queryset methods anytime you invoke them from the Car model: Car.objets.with... # try this
Question: I'm trying to close a frame on this page. What I want is to click in here:  It seems to be easy, but so far the following code (which should work) has failed: import selenium.webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC driver = selenium.webdriver.Chrome() driver.maximize_window() driver.get('https://www.bvc.com.co/variable-income-local-market/cemargos?tab=issuer-information') #X(close) bvc frame xpath = '//*[@id="__next"]/div/div[1]/div/div[1]/div/div/div/div/div[3]/div/div/div/div[3]/div[2]/span' class_name = 'sc-843139d2-14 iVPGqd' # Trying with XPath if 1: try: WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, xpath))).click() except: driver.find_element(By.XPATH, xpath).click() # Trying with class_name if 1: try: WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, class_name))).click() except: driver.find_element(By.CLASS_NAME, class_name).click()  The output using XPath: raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Stacktrace: #0 0x64a95375031a <unknown> ...  The output using class_name: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".sc-843139d2-14 iVPGqd"} 

Answer: when you do driver.find_element(By.CLASS_NAME, 'sc-843139d2-14 iVPGqd')  selenium translates (poorly, by adding one dot) your class name into a selector, so it becomes driver.find_element(By.CSS_SELECTOR, '.sc-843139d2-14 iVPGqd')  which means: find a iVPGqd tag (yes tag) contained within something with a class sc-843139d2-14. you don't want that. if you want to specify two classes for the same element you must collate them with no spaces between and use the proper locator CSS_SELECTOR driver.find_element(By.CSS_SELECTOR, '.sc-843139d2-14.iVPGqd')
Question: I have to read in data in Databricks Python/PySpark but the format is not the usual CSV or JSON so I have to iterate over a for loop. As a result it's very slow. The data looks like this, for millions of rows. It's not the same format each row, although there are certain common formats: HEADER0123 a bunch of spaces ACCTNUM999787666 more numbers ABC2XYZ some text So to parse I read from s3 as text and get the data by character position: raw_text = (spark.read .format("text") .option("mode", "PERMISSIVE") .option("header", "false") .option("inferSchema","false") .load(my_path)) my_list = [] input = raw_text.collect() for row in input: line = row[0].strip() header = line[0:6] acct = line[6:9] my_list.append(header, acct) etc.  Then later I create dataframes: df = spark.createDataFrame(my_list, "header string, acct int")  Even though I have experience with Spark dataframes this is the only way I can think of due to the unusual format. Is there a way to leverage Spark to process this kind of data? Or a way that doesn't require a for loop?

Answer: you're looking for substring()  my_list = [] input = raw_text.collect() for row in input: line = row[0].strip() header = line[0:6] acct = line[6:9] my_list.append(header, acct) df = spark.createDataFrame(my_list, "header string, acct int")  is same as df = ( raw_text .withColumn('header', F.substring('value', 0, 6)) .withColumn('acct', F.substring('value', 7, 3)) .drop('value') )   Also note that if each line in your input file is fixed length and has header and account fields separated by space then you can still read it as a csv. spark.read.option("delimiter", " ").csv(file) # or spark.read.csv(file, sep=' ')
Question: I've not been able to construct a pattern which can return an whole numbers that don't end in a sequence of digits. The numbers could be of any length, even single digits, but they will always be whole. Additionally, multiple numbers could be on the same line of text and I want to match them all. The numbers are always followed by either a single space or the end of the line or the end of the text. I'm matching in python 3.12 For example, over the text '12345 67890 123175 9876', let's say I want to get all numbers not ending in 175. I would want the following matches: 12345 67890 9876  I've tried using the following:  \d+(?<!175)(\b|$), which matched 3 empty strings,  text = "12345 67890 123175 9876" matches = findall(r"\d+(?<!175)(\b|$)", text) print(matches) > ['', '', '']   \d+(?!175)(\b|$), which matched 4 empty strings,  text = "12345 67890 123175 9876" matches = findall(r"\d+(?!175)(\b|$)", text) print(matches) > ['', '', '', '']   \d+(?<!175), which matched all 4 numbers  matches = findall(r"\d+(?<!175)", text) > ['12345', '67890', '12317', '9876']   \d+(?:175), which matched only the number ending in 175  matches = findall(r"\d+(?:175)", text) > ['123175'] 

Answer: You can use is a negative lookbehind .*(?<!a) that ensures the string does not end with a. \d++(?<!175)  Test here. Note that Possessive Quantifier (++) has been introduced in Python 3.11. Your 2nd approach from revision 1 was close, but not correct since the Greedy quantifier (+) would eat up all the digits, and then try to backtrack.
Question: I need to compute the percentage of positive values in the value column grouped by the group column. import polars as pl df = pl.DataFrame( { "group": ["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"], "value": [2, -1, 3, 1, -2, 1, 2, -1, 3, 2], } ) shape: (10, 2) ┌───────┬───────┐ │ group ┆ value │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═══════╡ │ A ┆ 2 │ │ A ┆ -1 │ │ A ┆ 3 │ │ A ┆ 1 │ │ A ┆ -2 │ │ B ┆ 1 │ │ B ┆ 2 │ │ B ┆ -1 │ │ B ┆ 3 │ │ B ┆ 2 │ └───────┴───────┘  In group A there are 3 out of 5 positive values (60%), while in column B there are 4 out 5 positive values (80%). Here's the expected dataframe. ┌────────┬──────────────────┐ │ group ┆ positive_percent │ │ --- ┆ --- │ │ str ┆ f64 │ ╞════════╪══════════════════╡ │ A ┆ 0.6 │ │ B ┆ 0.8 │ └────────┴──────────────────┘ 

Answer: You could use a custom group_by.agg with Expr.ge and Expr.mean. This will convert the values to False/True depending on the sign, then compute the proportion of True by taking the mean: df.group_by('group').agg(positive_percent=pl.col('value').ge(0).mean())  Output: ┌───────┬──────────────────┐ │ group ┆ positive_percent │ │ --- ┆ --- │ │ str ┆ f64 │ ╞═══════╪══════════════════╡ │ A ┆ 0.6 │ │ B ┆ 0.8 │ └───────┴──────────────────┘  Intermediates: ┌───────┬───────┬───────┬──────┐ │ group ┆ value ┆ ge(0) ┆ mean │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ bool ┆ f64 │ ╞═══════╪═══════╪═══════╪══════╡ │ A ┆ 2 ┆ true ┆ 0.6 │ # │ A ┆ -1 ┆ false ┆ 0.6 │ # group A │ A ┆ 3 ┆ true ┆ 0.6 │ # (True+False+True+True+False)/5 │ A ┆ 1 ┆ true ┆ 0.6 │ # = 3/5 = 0.6 │ A ┆ -2 ┆ false ┆ 0.6 │ # │ B ┆ 1 ┆ true ┆ 0.8 │ │ B ┆ 2 ┆ true ┆ 0.8 │ │ B ┆ -1 ┆ false ┆ 0.8 │ │ B ┆ 3 ┆ true ┆ 0.8 │ │ B ┆ 2 ┆ true ┆ 0.8 │ └───────┴───────┴───────┴──────┘
Question: How do I set environment variables in virtual environment creating by UV? I try setting it in .venv/scripts/activate_this.py and it doesn't work.

Answer: You can tell the uv run command to load environment variables from a file, either by using the --env-file option: uv run --env-file=.env myscript.py  Or by setting the UV_ENV_FILE environment variable: export UV_ENV_FILE=.env uv run myscript.py  You will find more details in the documentation.
Question: In the code below I'm creating a polars- and a pandas dataframe with identical data. I want to select a set of rows based on a condition on column A, then update the corresponding rows for column C. I've included how I would do this with the pandas dataframe, but I'm coming up short on how to get this working with polars. The closest I've gotten is by using when-then-otherwise, but I'm unable to use anything other than a single value in then. import pandas as pd import polars as pl df_pd = pd.DataFrame({'A': ['x', 'x', 'x', 'x', 'y', 'y', 'y', 'y'], 'B': [1, 1, 2, 2, 1, 1, 2, 2], 'C': [1, 2, 3, 4, 5, 6, 7, 8]}) df_pl = pl.DataFrame({'A': ['x', 'x', 'x', 'x', 'y', 'y', 'y', 'y'], 'B': [1, 1, 2, 2, 1, 1, 2, 2], 'C': [1, 2, 3, 4, 5, 6, 7, 8]}) df_pd.loc[df_pd['A'] == 'x', 'C'] = [-1, -2, -3, -4] df_pl ??? 

Answer: If you wrap the values in a pl.lit Series, you can index the values with Expr.get values = pl.lit(pl.Series([-1, -2, -3, -4])) idxs = pl.when(pl.col.A == 'x').then(1).cum_sum() - 1 df.with_columns(C = pl.coalesce(values.get(idxs), 'C'))  shape: (8, 3) ┌─────┬─────┬─────┐ │ A ┆ B ┆ C │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ x ┆ 1 ┆ -1 │ │ x ┆ 1 ┆ -2 │ │ x ┆ 2 ┆ -3 │ │ x ┆ 2 ┆ -4 │ │ y ┆ 1 ┆ 5 │ │ y ┆ 1 ┆ 6 │ │ y ┆ 2 ┆ 7 │ │ y ┆ 2 ┆ 8 │ └─────┴─────┴─────┘  These are the steps expanded. The indices are created, used to .get() and .coalesce() combines in the values from the other column. df.with_columns( idxs = idxs, values = values.get(idxs), D = pl.coalesce(values.get(idxs), 'C') )  shape: (8, 6) ┌─────┬─────┬─────┬──────┬────────┬─────┐ │ A ┆ B ┆ C ┆ idxs ┆ values ┆ D │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ i64 ┆ i32 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╪══════╪════════╪═════╡ │ x ┆ 1 ┆ 1 ┆ 0 ┆ -1 ┆ -1 │ │ x ┆ 1 ┆ 2 ┆ 1 ┆ -2 ┆ -2 │ │ x ┆ 2 ┆ 3 ┆ 2 ┆ -3 ┆ -3 │ │ x ┆ 2 ┆ 4 ┆ 3 ┆ -4 ┆ -4 │ │ y ┆ 1 ┆ 5 ┆ null ┆ null ┆ 5 │ │ y ┆ 1 ┆ 6 ┆ null ┆ null ┆ 6 │ │ y ┆ 2 ┆ 7 ┆ null ┆ null ┆ 7 │ │ y ┆ 2 ┆ 8 ┆ null ┆ null ┆ 8 │ └─────┴─────┴─────┴──────┴────────┴─────┘
Question: I have a numpy array subclass, and I'd like to be able to concatenate them. import numpy as np class BreakfastArray(np.ndarray): def __new__(cls, n=1): dtypes=[("waffles", int), ("eggs", int)] obj = np.zeros(n, dtype=dtypes).view(cls) return obj b1 = BreakfastArray(n=1) b2 = BreakfastArray(n=2) con_b1b2 = np.concatenate([b1, b2]) print(b1.__class__, con_b1b2.__class__)  this outputs <class '__main__.BreakfastArray'> <class 'numpy.ndarray'>, but I'd like the concatenated array to also be a BreakfastArray class. It looks like I probably need to add a __array_finalize__ method, but I can't figure out the right way to do it.

Answer: Expanding simon's solution, this is what I settled on so other numpy functions fall-back to standard ndarray (so, numpy.unique(b2["waffles"]) works as expected). Also a slight change to concatenate so it will work for any subclasses as well. import numpy as np HANDLED_FUNCTIONS = {} class BreakfastArray(np.ndarray): def __new__(cls, *args, n=1, **kwargs): dtypes=[("waffles", int), ("eggs", int)] obj = np.zeros(n, dtype=dtypes).view(cls) return obj def __array_function__(self, func, types, args, kwargs): # If we want "standard numpy behavior", # convert any BreakfastArray to ndarray views if func not in HANDLED_FUNCTIONS: new_args = [] for arg in args: if issubclass(arg.__class__, BreakfastArray): new_args.append(arg.view(np.ndarray)) else: new_args.append(arg) return func(*new_args, **kwargs) if not all(issubclass(t, BreakfastArray) for t in types): return NotImplemented return HANDLED_FUNCTIONS[func](*args, **kwargs) def implements(numpy_function): def decorator(func): HANDLED_FUNCTIONS[numpy_function] = func return func return decorator @implements(np.concatenate) def concatenate(arrays): result = arrays[0].__class__(n=sum(len(a) for a in arrays)) return np.concatenate([np.asarray(a) for a in arrays], out=result)
Question: I'm trying to find a better/faster way to do this. I have a rather large dataset (~200M rows) with individual dates per row. I want to find the average yearly rate per group from 2018 to 2019. I know I could create a small df with the results and merge it back in but, I was trying to find a way to use transform. Not sure if it would just be faster to merge. Extra points for one-liners. Sample data rng = np.random.default_rng(seed=123) df = pd.DataFrame({'group':rng.choice(list('ABCD'), 100), 'date':[(pd.to_datetime('2018')+pd.Timedelta(days=x)).normalize() for x in rng.integers(0, 365*5, 100)], 'foo':rng.integers(1, 100, 100), 'bar':rng.integers(50, 200, 100)}) df['year'] = df['date'].dt.year  This works #find average 2018 and 2019 'foo' and 'bar' for col in ['foo', 'bar']: for y in [2018, 2019]: df[col+'_'+str(y)+'_total'] = df.groupby('group')['year'].transform(lambda x: df.loc[x.where(x==y).dropna().index, col].sum()) #find 2018 and 2019 rates for y in [2018, 2019]: df['rate_'+str(y)] = df['foo_'+str(y)+'_total'].div(df['bar_'+str(y)+'_total']) #find average rate df['2018_2019_avg_rate'] = df[['rate_2018', 'rate_2019']].mean(axis=1)  Thing's I've tried that don't quite work (I'm using apply to test if it works before switching to transform) #gives yearly totals for each year and each column, but further 'apply'ing to find rates then averaging isn't working after I switch to transform df.groupby(['group', 'year'])['year'].apply(lambda x: df.loc[x.where(x.between(2018, 2019)).dropna().index, ['foo', 'bar']].sum()) #close but is averaging too early df.groupby(['group', 'year'])['year'].apply(lambda x: df.loc[i, 'foo'].sum()/denom if (denom:=df.loc[i:=x.where(x.between(2018, 2019)).dropna().index, 'bar'].sum())>0 else np.nan) 

Answer: You can't perform multiple filtering/aggregations efficiently with a groupby.transform. You will have to loop. A more efficient approach would be to combine a pivot_table + merge: cols = ['foo', 'bar'] years = [2018, 2019] tmp = (df[df['year'].isin(years)] .pivot_table(index='group', columns='year', values=cols, aggfunc='sum') [cols] .pipe(lambda x: x.join(pd.concat({'rate': x['foo'].div(x['bar'])}, axis=1))) ) avg_rate = tmp['rate'].mean(axis=1) tmp.columns = tmp.columns.map(lambda x: f'{x[0]}_{x[1]}_total') tmp[f'{"_".join(map(str, years))}_avg_rate'] = avg_rate out = df.merge(tmp, left_on='group', right_index=True)  Output:  group date foo bar year foo_2018_total foo_2019_total bar_2018_total bar_2019_total rate_2018_total rate_2019_total 2018_2019_avg_rate 0 A 2022-03-11 59 91 2022 343 270 972 875 0.352881 0.308571 0.330726 1 C 2018-08-22 56 52 2018 175 325 331 902 0.528701 0.360310 0.444506 2 C 2019-04-24 47 89 2019 175 325 331 902 0.528701 0.360310 0.444506 3 A 2019-04-16 43 102 2019 343 270 972 875 0.352881 0.308571 0.330726 4 D 2019-11-25 3 56 2019 126 222 224 696 0.562500 0.318966 0.440733 5 A 2018-01-06 86 148 2018 343 270 972 875 0.352881 0.308571 0.330726 ... 99 B 2018-02-25 32 90 2018 253 204 703 400 0.359886 0.510000 0.434943
Question: It seems that a pymongo 4.10 async client does not raise an exception when there is a problem with the connection. Taken from the doc, a test without any mongo DB running locally yields: >>> import asyncio >>> from pymongo import AsyncMongoClient >>> client = AsyncMongoClient('mongodb://localhost:27017/') >>> asyncio.run(client.aconnect()) # no errors  When activating debug logs I see the connection being refused but I would expect an exception to be raised. >>> import logging >>> logging.basicConfig(level='DEBUG') >>> asyncio.run(client.aconnect()) DEBUG:asyncio:Using selector: KqueueSelector DEBUG:pymongo.topology:{"topologyId": {"$oid": "676020be62e71d3fe6f27721"}, "serverHost": "localhost", "serverPort": 27017, "awaited": false, "durationMS": 2.786167000522255, "failure": "\"AutoReconnect('localhost:27017: [Errno 61] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')\"", "message": "Server heartbeat failed"}  I would expect the DEBUG log error to be an exception. Am I misunderstanding something with the async client ?

Answer: The mongo client uses connection pools etc in the background, even though you tell it to explicitly connect (why?) it doesn't raise an exception for failing to connect until you actually try to read or write from/to the DB. But you can check if/where it's connected: >>> list(client.nodes) [('10.0.0.1', 27017)]  The result will be an empty list if aconnect fails. But if you try any communication such as: >>> await client.server_info()  ... you will get an exception: pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 1.9985503089847043s, Topology Description: <TopologyDescription id: 6760281eda9a9980ea35e425, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>  The pymongo async driver is just built like this.. why the tutorial tells you to use aconnect() ... I have no idea. I didn't even know it existed. Btw, you can use python -m async to get a REPL where you can run async commands without asyncio.run()
Question: I have a Pandas DataFrame that looks like: ID result 1 [.1,.5] 2 [.4,-.2,-.3,.1,0] 3 [0,.1,.6]  How can split this column of lists into two columns? Desired result: ID result_1 result_2 result_3 result_4 result_5 1 .1 .5 NaN NaN NaN 2 .4 -.2 -.3 .1 0 3 0 .1 .6 NaN NaN  I have digged into it a little and found this: Split a Pandas column of lists into multiple columns but this only seems to apply to list with a constant number of elements. Thank you so much in advance.

Answer: You can do this as suggested in linked post. import pandas as pd # your example code data = {"ID": [1, 2, 3], "result": [[0.1, 0.5], [0.4, -0.2, -0.3, 0.1, 0], [0, 0.1, 0.6]]} df = pd.DataFrame(data) print(df)  answer out = df[['ID']].join( pd.DataFrame(df['result'].tolist()) .rename(columns=lambda x: f'result_{x + 1}') )  out:  ID result_1 result_2 result_3 result_4 result_5 0 1 0.1 0.5 NaN NaN NaN 1 2 0.4 -0.2 -0.3 0.1 0.0 2 3 0.0 0.1 0.6 NaN NaN
Question: I am making a simple server in python using http.server package. My goal is to log the data of POST from client to server. The problem I am having is rfile.read() is delaying execution until next POST request or if the connection is disconnected. However this problem doesn't occur if the length of the content is specified as the argument in rfile.read(). The below is the subclass for BaseHTTPRequestHandler of http.server. class Reqhand(BaseHTTPRequestHandler): def do_POST(self): self.send_response(200) #The lines below don't execute until the next POST req req = self.rfile.read() print(req)  The below code executes as intended in a single POST request. class Reqhand(BaseHTTPRequestHandler): def do_POST(self): self.send_response(200) content_len = int(self.headers.get_all('Content-Length')[0]) req = self.rfile.read(content_len) print(req) 

Answer: This happens because rfile.read() waits indefinitely until the client closes the connection or signals the end of the data stream. This behavior is by design when the Content-Length header is not provided, as the server does not know how much data to expect. To handle this, you should always rely on the Content-Length header in HTTP post requests to determine how much data to read.  Make sure the client sends the Content-Length header in its post request.
Question: I have the following training code. I am quite sure I call loss.backward() just once, and yet I am getting the error from the title. What am I doing wrong? Note that the X_train_tensor is output from another graph calculation, so it has required_grad=True as you can see in the print statement. Is this the source of the problem, and if so, how can I change it? It won't allow me to toggle it directly on the tensor. for iter in range(max_iters): start_ix = 0 loss = None while start_ix < len(X_train_tensor): loss = None end_ix = min(start_ix + batch_size, len(X_train_tensor)) out, loss, accuracy = model(X_train_tensor[start_ix:end_ix], y_train_tensor[start_ix:end_ix]) # every once in a while evaluate the loss on train and val sets if (start_ix==0) and (iter % 10 == 0 or iter == max_iters - 1): out_val, loss_val, accuracy_val = model(X_val_tensor, y_val_tensor) print(f"step {iter}: train loss={loss:.2f} train_acc={accuracy:.3f} | val loss={loss_val:.2f} val_acc={accuracy_val:.3f} {datetime.datetime.now()}") optimizer.zero_grad(set_to_none=True) print (iter, start_ix, X_train_tensor.requires_grad, y_train_tensor.requires_grad, loss.requires_grad) loss.backward() optimizer.step() start_ix = end_ix + 1  This is the error: RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.  Update: this is where the model input tensors are coming from, as output of other (autoencoder) model: autoencoder.eval() with torch.no_grad(): # it seems like adding this line solves the problem? X_train_encoded, loss = autoencoder(X_train_tensor) X_val_encoded, loss = autoencoder(X_val_tensor) X_test_encoded, loss = autoencoder(X_test_tensor)  Adding the with torch.no_grad() line above solves the issue, but I don't understand why. Does it actually change how the outputs are generated, how does that work?

Answer: From what I understand, the X_train_tensor is output from the autoencoder. When you do not run torch.no_grad() during the encoding step, a computational graph is created for the outputs of the autoencoder, which links the autoencoder's operations and weights to the encoded tensors. In your code, since the model's output uses the X_train_tensor, the model's loss is connected to the autoencoder's computational graph. When you call loss.backward() the first time, PyTorch traverses the entire computational graph, including the autoencoder, to compute gradients and then clears the graph. When you call loss.backward() in the second iteration of the loop, you are attempting to traverse the cleared autoencoder's computational graph. torch.no_grad() prevents PyTorch from creating the autoencoder computational graph or linking the resulting loss to the autoencoder.