eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Problem I am running a LightGBMModel via Darts with some (future) covariates. I want to understand the relevance of the different (lagged) features. In particular, I would like to retrieve the feature importance for the lagged target variable as well as for the covariates using the original column names from the Darts TimeSeries object. In the LightGBM model object after fitting I can only see generic column names ("column_0", "column_1"). How can I connect this to meaningful names (e.g., target_lag_1, target_lag_2, name_of_covariate_lag_1, ...). I want to include several future covariates (e.g., several datetime attributes like day of week with different encodings). It does not matter where the datetime attributes are created (e.g., using pandas, using Darts itself). Minimal reproducable example I adopted the example from the documentation This is the code from the documentation, just setting up the data and fitting the model: from darts.datasets import WeatherDataset from darts.models import LightGBMModel series = WeatherDataset().load() # predicting atmospheric pressure target = series['p (mbar)'][:100] # optionally, use past observed rainfall (pretending to be unknown beyond index 100) past_cov = series['rain (mm)'][:100] # optionally, use future temperatures (pretending this component is a forecast) future_cov = series['T (degC)'][:106] # predict 6 pressure values using the 12 past values of pressure and rainfall, as well as the 6 temperature # values corresponding to the forecasted period model = LightGBMModel( lags=12, lags_past_covariates=12, lags_future_covariates=[0,1,2,3,4,5], output_chunk_length=6, verbose=-1 ) model.fit(target, past_covariates=past_cov, future_covariates=future_cov)  Having fitted the model, I now want to analyze the importance of the features. for i, estimator in enumerate(model.model.estimators_): print(f"Target {i} Importance (Gain):") # Access LightGBM booster booster = estimator.booster_ # Get feature names feature_names = booster.feature_name() # Get gain-based importance importance = booster.feature_importance(importance_type='gain') # Create mapping named_importance = dict(zip(feature_names, importance)) print(named_importance)  This returns the feature importance for several columns in each estimator. But the feature names are generic names generated by LightGBM ('Column_1', 'Column_2', ...). I do not know how to link this back to the original column names in the TimeSeries object from Darts (e.g., 'rain (mm)', ''T (degC)') with the additional information which lag a feature importance is referring to.

Answer: The features that go into the models are available in model.lagged_feature_names. One of the authors addressed feature importances in Issue#1826, doing mostly what you've done, but they also referenced that along with a note about the feature names in Issue#2125.
Question: So, my question is regarding a code that looks like this: def f(condition: bool) -> int | None: if condition: return 1 def g(condition: bool) -> int | None: if condition: return return 1  This is clearly valid python code, the idea is that the function will try to do something, if it succeeds it will return the result, but if it fails will return None My problem is that mypy complains with the following error: 1: error: Missing return statement [return] 7: error: Return value expected [return-value]  Now, python will always implicitly return None when a function is missing a return statement, which is what I am using in my function. My question is, is there any option in mypy to (with minimal or no changes to the code) tell it to stop complaining about this valid code? To be clear, I would still want it to complain about this code: def f(condition: bool) -> int: if condition: return 1 def g(condition: bool) -> int: if condition: return return 1 

Answer: Provide an explicit return statement when the if statement does not match in the first function and explicitly return None in the second function: def f(condition: bool) -> int | None: if condition: return 1 return None def g(condition: bool) -> int | None: if condition: return None return 1  fiddle  If you want to silence the first error without changing the code then you can use the --no-warn-no-return option. If you want to locally silence the return error on line 1 and the return-value error on line 7 then you can use: def f(condition: bool) -> int | None: # type: ignore[return] if condition: return 1 def g(condition: bool) -> int | None: if condition: return # type: ignore[return-value] return 1  (But it is less to type to provide explicit None return values.)
Question: Can you share some ideas of how to find curve points (orange color marked places) like shown in picture:  I've tried this code: result = [] for i in range(len(df)): if i == 0 or df['y'].iloc[i] != df['y'].iloc[i - 1]: result.append(df.iloc[i]) continue if i < len(df) - 1 and df['y'].iloc[i] != df['y'].iloc[i + 1]: result.append(df.iloc[i])  but the problem is, my 'y' isn't always equal between breakpoints and returns too many values. Could you give some advice, how to achieve my goal? Used data for picture: time = np.arange(0, 2200, 100) values = np.array([-0.1, 0, 0.13, 0.27, 0.27, 0.4, 0.27, 0.27, 0.13, 0.13, 0.01, 0.01, -0.13, -0.13, -0.27, -0.4, -0.4, -0.27, -0.13, -0.13, 0, 0]) full_time = np.arange(0, 2200, 1) full_values = np.interp(full_time, time, values) 

Answer: Here is the full code: import numpy as np import matplotlib.pyplot as plt from scipy.signal import argrelextrema time = np.arange(0, 2200, 100) values = np.array([ -0.1, 0, 0.13, 0.27, 0.27, 0.4, 0.27, 0.27, 0.13, 0.13, 0.01, 0.01, -0.13, -0.13, -0.27, -0.4, -0.4, -0.27, -0.13, -0.13, 0, 0 ]) x = np.arange(0, 2200, 1) y = np.interp(x, time, values) dy = np.gradient(y) d2y = np.gradient(dy) curvature = np.abs(d2y) / (1 + dy**2)**1.5 threshold = 0.0005 corner_indices = np.where(curvature > threshold)[0] filtered_corners = [corner_indices[0]] for idx in corner_indices[1:]: if idx - filtered_corners[-1] > 10: filtered_corners.append(idx) filtered_corners = [0] + filtered_corners + [len(x) - 1] filtered_corners = sorted(set(filtered_corners)) plt.figure(figsize=(12, 6)) plt.plot(x, y, label='Interpolated Curve') plt.plot(x[filtered_corners], y[filtered_corners], 'o', color='orange', label='Corners') plt.xlabel('x') plt.ylabel('y') plt.grid(True) plt.legend() plt.tight_layout() plt.show()  Output:
Question: I need to format np.float64 floating values without leading digits before the dot, for example -2.40366982307 as -.240366982307E+01, in python. This is to allow me to write in RINEX 3.03 the values with 4X, 4D19.12 formats. I have tried f"{x:.12E}" but it always has a leading 1 for numbers greater than 1. I have also tried np.format_float_positional and np.format_float_scientific but those don't have restrictions on leading digits.

Answer: I would go for something custom designed from scratch. I agree, however, I did not think about all the marginal cases that can occur. # Format a floating-point number to RINEX format def format_rinex(value): if not np.isfinite(value): return f"{value}" sign = '-' if value < 0 else ' ' abs_value = np.abs(value) exponent = 1 + np.int32(np.floor(np.log10(abs_value))).item() if value != 0 else 0 mantissa = abs_value / (10 ** exponent) mantissa_str = f'{mantissa:.12f}'.replace("0.", ".") return f'{sign}{mantissa_str}E{exponent:+03d}'  Formats your example number into -.240366982307E+01.
Question: I am trying to access the login iframe from https://www.steelmarketupdate.com/. Previously I was able to access this via XPATH client.switch_to.frame(client.find_element(By.XPATH, "/html/body/div[6]/div/iframe"))  However this is no longer working. I found the length of all iframe elements to be 6, and I am unable to access any of these even via the index location. How can I switch to this frame?

Answer: Try this: # This relative XPath expression locates the <iframe> element which contains value piano in the ID attribute By.XPATH, "//iframe[contains(@id,'piano')]"  or this: # This is an XPath expression which locates the 3rd <iframe> element from top of the DOM By.XPATH, "(//iframe)[3]"  Full line of code: client.switch_to.frame(client.find_element(By.XPATH, "//iframe[contains(@id,'piano')]"))  UPDATE: Full code with selenium's waits to effectively locate elements. Code explaination in comments. import time from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By driver = webdriver.Chrome() driver.get("https://www.steelmarketupdate.com/") driver.maximize_window() wait = WebDriverWait(driver, 10) # click on Log In button wait.until(EC.element_to_be_clickable((By.XPATH, "(//li[@id='pp-subs-login'])[1]"))).click() # enter inside IFRAME wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[contains(@id,'piano')]"))) # send keys to email and password fields wait.until(EC.element_to_be_clickable((By.NAME, "email"))).send_keys("testEmail") wait.until(EC.element_to_be_clickable((By.XPATH, "//input[@aria-label='password']"))).send_keys("testPassword") # switch to default content driver.switch_to.default_content() # wait for a while to observe the result time.sleep(10)  Result:
Question: I have a large Excel spreadsheet. I'm only interested in certain columns. Furthermore, I'm only interested in rows where specific columns meet certain criteria. The following works: import pandas as pd import warnings # this suppresses the openpyxl warning that we're seeing warnings.filterwarnings("ignore", category=UserWarning, module="openpyxl") # These are the columns we're interested in COLUMNS = [ "A", "B", "C" ] # the source file XL = "source.xlsx" # sheet name in the source file SHEET = "Sheet1" # the output file OUTPUT = "target.xlsx" # the sheet name to be used in the output file OUTSHEET = "Sheet1" # This loads the entire spreadsheet into a pandas dataframe df = pd.read_excel(XL, sheet_name=SHEET, usecols=COLUMNS).dropna() # this replaces the original dataframe with rows where A contains "FOO" df = df[df["A"].str.contains(r"\bFOO\b", regex=True)] # now isolate those rows where the B contains "BAR" df = df[df["B"].str.contains(r"\bBAR\b", regex=True)] # output to the new spreadsheet df.to_excel(OUTPUT, sheet_name=OUTSHEET, index=False)  This works. However, I can't help thinking that there might be a better way to manage the selection criteria especially if / when they get more complex. Or is it a case of "step-by-step" is good?

Answer: You can certainly chain all your commands to avoid using intermediate variables, and combine all filters into a single expression (for example defining the condition in a col:regex dictionary and using loc with numpy.logical_and.reduce): conditions = {'A': r'\bFOO\b', 'B': r'\bBAR\b'} (pd.read_excel(XL, sheet_name=SHEET, usecols=COLUMNS).dropna() .loc[lambda x: np.logical_and.reduce([x[col].str.contains(cond, regex=True) for col, cond in conditions.items()])] .to_excel(OUTPUT, sheet_name=OUTSHEET, index=False) )  Alternative with a custom filtering function: def cust_filter(df): m1 = df['A'].str.contains(r'\bFOO\b', regex=True) m2 = df['B'].str.contains(r'\bBAR\b', regex=True) return df[m1 & m2] (pd.read_excel(XL, sheet_name=SHEET, usecols=COLUMNS).dropna() .pipe(cust_filter) .to_excel(OUTPUT, sheet_name=OUTSHEET, index=False) )  Example input:  A B C 0 ABC GHI other 1 FOO BAR other 2 FOO JKL other 3 DEF BAR other  Example output:  A B C 1 FOO BAR other
Question: I am puzzled by an inconsistency when calling .strftime() for dates which are pre-1000 AD, using Python's datetime module. Take the following example: import datetime old_date = datetime.date(year=33, month=3, day=28) # 28th March 33AD old_date.isoformat() >>> "0033-03-28" # Fine! old_date.strftime("%Y-%m-%d") >>> "33-03-28" # Woah - where did my leading zeros go? # And even worse datetime.datetime.strptime(old_date.strftime("%Y-%m-%d"), "%Y-%m-%d") >>> ... File "<input>", line 1, in <module> File "/usr/lib/python3.12/_strptime.py", line 554, in _strptime_datetime tt, fraction, gmtoff_fraction = _strptime(data_string, format) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/_strptime.py", line 333, in _strptime raise ValueError("time data %r does not match format %r" % ValueError: time data '33-03-28' does not match format '%Y-%m-%d'  The documentation shows examples of %Y yielding zero-padded years. Even using %G, which is documented to be an ISO-8601 4-digit year, is showing only two digits. This caused a problem in an application where a user can enter a date, and if they type in an old date the exception above would arise when trying to convert a date-string back into a date. Presumably there is something in my local configuration which is causing this, as this seems too obvious to be a bug in Python. I'm using Python 3.12 on Ubuntu 24.04.

Answer: This is caused by the implementation of .strftime() in the C library in Linux omitting any leading zeros from %Y and %G. The related issue in CPython's issue tracker is here. Thanks to jonrsharpe's comment for the answer, and highlighting this section of the documentation:  "The full set of format codes supported varies across platforms, because Python calls the platform C library’s strftime() function, and platform variations are common."
Question: I aim to plot a contour plot of flux, but instead of closed contour curves, plt.contour() returns a series of lines with the same height. Psi is defined as a np.array and has a 320 by 200 shape. fig, ax = plt.subplots() r_end = grid_start[0] + grid_step[0] * grid_size[0] z_end = grid_start[1] + grid_step[1] * grid_size[1] X = np.arange(grid_start[0], r_end, grid_step[0]) # shape (200,) Y = np.arange(grid_start[1], z_end, grid_step[1]) # shape (320,) x, y = np.meshgrid(X, Y) CS = ax.contour(x, y, Psi) ax.clabel(CS, fontsize=10)  The result looks very weird: all contours image I also created contour plots with some levelling for enhanced visibility. contour plot, level 0 contour plot, level 25 The desired result is smth like this: desired Psi Contours If I reshape the given array to (3200,20) the periodicity is lost, but it is still weird. reshaped result How can I fix this issue? Thanks for your help.

Answer: I think that your original data is a different size: maybe 3200 by 20, not 320 by 200. (You should check). The datafile does have 320 rows of 200 columns, but I suspect that is an artefact: maybe they were simply limited by line length. If I reshaped to 3200 by 20 then this is what I get:   import numpy as np import matplotlib.pyplot as plt NX, NY = 320, 200 psi = np.zeros( ( NX, NY ) ) with open( 'flux_test.txt' , 'r' ) as input: for row, line in enumerate( input ): for col, str in enumerate( line.split() ): psi[row,col] = float( str ) psi = psi.reshape( 3200, 20 ) fig, ax = plt.subplots(figsize=[10, 6]) CS = ax.contour( psi ) plt.show()  Interestingly, if I reshape to 200 by 320 then I get the following which, I suppose, might also be right. I don't know what you are trying to get as an end result. Note that both Fortran and Matlab use column-major array storage order - the opposite of Python's numpy - so I still think you need to go back to how the data file was created. You aren't giving us that information.
Question: A colleague and I were reviewing some student submissions. He likes using IDLE, while I use PyCharm. The student developed their code in PyCharm. A simplified example of the students work is: file = open('test_file.txt','w') file.write('This is a test file.') print('Completed') exit()  The student has made an error in their file handling and created a situation where the file is left open after the code is completed. When run from within PyCharm the file.write is completed and the file is updated as expected. When run from within IDLE the file.write appears not to be completed and the file is empty. When run from the command line (btw, we all use macbooks) the file.write is completed and the file has the line of text. One more clue is that when run within IDLE after 'Completed' is output, there is a system dialog that states, "Your program is still running! Do you want to kill it?". This warning does not appear when run from the command line or within PyCharm. We are trying to understand the difference between these behaviours given that we believe they are all doing the same process of invoking the same interpreter.

Answer: IDLE leaves the interpreter running after executing the code. You can go back to the Shell and inspect variables, for example. If you exit idle or restart the shell (Ctrl-F6) the interpreter exits (or restarts) and the file will be flushed and closed. Without restarting the shell, the file will still be open and cached writes may not have been written to disk yet. The other IDEs appear to exit the interpreter when the script completes. It's a design decision by the developers. You would get a similar result from the command line by running python -i script.py, which leaves the interpreter running after executing script.py. The file won't be flushed until exiting. FYI: The exit() is causing the IDLE system message. It kills the interpreter and closes the Shell, but since it is a kill and not a clean exit the file isn't flushed.
Question: Is it possible? Or it seems that each sheet is a separate environment? CONTEXT: A clean way to read 200 rows of data (and 30+ columns) is using something like df=xl("A:BS", headers=True) So a user wants a filtered view of my data on sheet2. e.g., df[df['project'] =='bench'] (and despite that one can use Excel filter but prefers not to. E.g., rewrite in python-in-excel R markdown logic that parses input from excel as html page with 30+ analyses and TOC and graphs and headings)

Answer: You can do something like import pandas as pd df = Excel("Data!A:AD", headers=True) filtered_df = df[df['project'] == 'bench'] filtered_df
Question: I created a programm on python to read serial data from an arduino and show it on a tkinter window. I made a Thread to read the from the arduino and the tkinter programm. My Programm runs perfectly on windows but the problem is that i want to use it on my ubuntu laptop and there it doesn't work. It works ok but there are some problems. I have some sensor values that update if some new information comes from the arduino, but the values are not shown. Also and most importend no buttons work and I can't close the window. My Programm has this structure: |--Main.py (for starting the programm) |--App.py (for the application) | |--arduino | | |--serial_reader.py (the Thread to read from my arduino) | | |--Input_manager.py and save_Input.py (for saving the data in variebles) | |--managers | | |--button_manager.py | | |--image_manager.py | | |--text_manager.py (to manage the widgets of the ui) | |--ui | | |--custom_button.py | | |--custom_image.py | | |--custom_text.py (for creating/deleting/updating the widgets)  I tried narrowing down of the problem and the programm worked perfectly fine without this funktion in the custom_text class. def delete_Text(self): if self.anzahl == 1: self.canvas.delete(self.tag) else: for i in range(0, self.anzahl): self.canvas.delete(self.tag + str(i))  I don't see why this crashes the entire window. In the app I have a loop with the after funktion in tkinter: def loop(self): try: while not data_queue.empty(): data = data_queue.get_nowait() self.serial_Reader.read_Serial_Values(data) self.text_manager.update() except queue.Empty: print("leer") pass self.window.after(1, self.loop)  It looks if the Thread got some information, then it will save this information in specific Variables and then the text should update. In the text_manager the update function just calls th update function in the custom_ text file. which looks like this: def update_Text(self): self.delete_Text() self.create_Text()  And that starts the delete_Text function which crashes the programm. I don't understand what is wrong with this function. I mean it worked and it doesn't show any error idk, but it's logic that the text is not updating, because I update the text with deleting and creating a new text. Hopfully somebody can help me

Answer: The problem probably isn’t with your delete_Text() function itself, but more with how you're using threads with tkinter. In tkinter, you can’t call methods like canvas.delete, canvas.create_text, label.config, and so on, from a thread other than the main thread. I had a similar issue myself — everything worked fine on Windows, but on Linux, tkinter behaved differently, like it was stricter or something. And yeah, the app would freeze or the window just wouldn’t respond, exactly like what you’re seeing. So, most likely, the issue is thread-related, not with the logic of your function.
Question: I have a PySpark UDF which when I try to apply to each row for one of the df columns and get a new column, I get a [Ljava.lang.Object;@7e44638d (different value after the @ for each row) Please see the udf below: def getLocCoordinates(property_address): url = "https://maps.googleapis.com/maps/api/geocode/json" querystring = {f"address":property_address},"key":"THE_API_KEY"} response = requests.get(url, params=querystring) response_json = json.loads(response.text) for adr in response_json['results']: geometry = adr['geometry'] coor = geometry['location'] lat = coor['lat'] lng = coor['lng'] coors = lat, lng return coors getCoorsUDF = udf(lambda x:getLocCoordinates(x)) df = df.withColumn("AddressCoordinates", getCoorsUDF(col("FullAddress") ) )  I tried:  getCoorsUDF = udf(getLocCoordinates, FloatType()) --> returns NULL for each row of the newly create "AddressCoordinates" column.  getCoorsUDF = udf(getLocCoordinates, StringType()) --> returns [Ljava.lang.Object;@  getCoorsUDF = udf(getLocCoordinates) --> returns [Ljava.lang.Object;@   The result looks like so:    Ref Num FullAddress AddressCoordinates     1234 Some Address [Ljava.lang.Object;@    This gets returned for each row in the dataframe. Initially I was using the function in a Python notebook and it was working fine, lat and lng was returning for each adress. However, I had to move this to PySpark and I am hitting a brick wall here.

Answer: I think that you're seeing the [Ljava.lang.Object;@... output because your UDF is returning a Python tuple ((lat, lng)), and PySpark doesn't know how to serialize that into a DataFrame column unless you explicitly define a return schema that Spark understands. You should return a StructType with fields for lat and lng. For example you can do something like this: from pyspark.sql.functions import udf, col from pyspark.sql.types import StructType, StructField, DoubleType import requests import json # defining return type for the UDF location_schema = StructType([ StructField("lat", DoubleType(), True), StructField("lng", DoubleType(), True) ]) def getLocCoordinates(property_address): url = "https://maps.googleapis.com/maps/api/geocode/json" params = { "address": property_address, "key": "YOUR_API_KEY" } try: response = requests.get(url, params=params) data = response.json() if data['results']: location = data['results'][0]['geometry']['location'] return {"lat": location['lat'], "lng": location['lng']} except Exception as e: print(f"Error: {e}") return None # registering the UDF with schema getCoorsUDF = udf(getLocCoordinates, location_schema) # now you apply the UDF df = df.withColumn("AddressCoordinates", getCoorsUDF(col("FullAddress"))) # an option would be to extract lat and lng as separate columns df = df.withColumn("Latitude", col("AddressCoordinates.lat")) \ .withColumn("Longitude", col("AddressCoordinates.lng"))