eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I would like to define a Snowpark UDF with input type snowflake.snowpark.Row. The reason for this is that I would like to mimic the pandas.apply approach where I can define my business logic in some class, and then apply the logic to each row of the Snowpark dataframe. Each column can be easily mapped to a class attribute with asDict For example (running from the Snowflake Python worksheet): import snowflake.snowpark as snowpark from snowflake.snowpark.functions import udf from snowflake.snowpark import Row from snowflake.snowpark.types import IntegerType from dataclasses import dataclass @dataclass class MyEvent: attribute1: str = 'dummy' attribute2: str = 'unknown' def someCalculation(self) -> int: return len(self.attribute1) + len(self.attribute2.strip()) def testSomeCalculation(): inputDict = {'attribute1': 'foo', 'attribute2': 'baz'} event = MyEvent(**inputDict) print(event.someCalculation()) def main(session: snowpark.Session): some_logic = udf(lambda row: MyEvent(**(row.asDict())).someCalculation() , return_type=IntegerType() , input_types=[Row])  However, when I try to use snowpark.Row as input type, I get an unsupported data type: File "snowflake/snowpark/_internal/udf_utils.py", line 972, in create_python_udf_or_sp input_sql_types = [convert_sp_to_sf_type(arg.datatype) for arg in input_args] File "snowflake/snowpark/_internal/udf_utils.py", line 972, in <listcomp> input_sql_types = [convert_sp_to_sf_type(arg.datatype) for arg in input_args] File "snowflake/snowpark/_internal/type_utils.py", line 195, in convert_sp_to_sf_type raise TypeError(f"Unsupported data type: {datatype.__class__.__name__}") TypeError: Unsupported data type: type  I see that all the UDF examples use basic types from snowpark.types. Is there any fundamental reason why the input type cannot be a snowpark.Row ? I know I could list explicitly all MyEvent attributes in input_type=[], but that is going to be error prone and defeating the purpose of designing my code around a class representing my business object.

Answer: Solution tested in Snowflake Python worksheet, based on the suggestion by @felipe-hoffa above import snowflake.snowpark as snowpark from snowflake.snowpark.functions import col, udf from snowflake.snowpark.types import IntegerType, VariantType from dataclasses import dataclass import json def main(session: snowpark.Session): @dataclass class MyEvent: attribute1: str = 'dummy' attribute2: str = 'unknown' def someText(self) -> str: return f"someText {len(self.attribute1)} : {self.attribute1=}, {self.attribute2=}" def wrap_some_text(x) -> str: return MyEvent(**json.loads(x)).someText() my_event_get_text = udf(lambda x: wrap_some_text(x), return_type=VariantType(), input_types=[VariantType()]) df = session.create_dataframe(['{"attribute1":"value1", "attribute2":"value20"}']).to_df("col1") df = df.select(col("col1"), my_event_get_text(col("col1").astype("variant")).as_("my_event_get_text") ).show() return df
Question: I have this code on google colab which allows me to optimise an LSTM model using gridsearchCV, but recently an error message has appeared: ModuleNotFoundError: No module named 'keras.wrappers'. is there another module other than 'keras.wrappers' that allows the code to be restarted? Code: from keras.layers import Dense, LSTM, Dropout from keras import optimizers from sklearn.model_selection import GridSearchCV from keras.wrappers.scikit_learn import KerasRegressor def create_model(unit, dropout_rate, lr ): model=Sequential() model.add(LSTM(unit,return_sequences=True, input_shape=(1,5))) model.add(Dropout(dropout_rate)) model.add(LSTM(unit)) model.add(Dropout(dropout_rate)) model.add(Dense(1)) adam= optimizers.Adam(lr) model.compile(optimizer=adam, loss='mean_squared_error') return model my_regressor = KerasRegressor(build_fn=create_model, verbose=2) grid_param_LSTM = { 'unit': [50, 70, 120], 'batch_size': [12, 24, 48], 'epochs': [200], 'lr': [0.001, 0.01, 0.1], 'dropout_rate':[0.1, 0.2, 0.3] } grid_GBR = GridSearchCV(estimator=my_regressor, param_grid = grid_param_LSTM, scoring = 'neg_root_mean_squared_error', cv = 2) grid_GBR.fit(X_train, y_train) print("Best: %f using %s" % (grid_GBR.best_score_, grid_GBR.best_params_)) 

Answer: This works for me pip install keras==2.12.0  Another Approach you can try pip uninstall tensorflow pip install tensorflow==2.12.0
Question: I tried running pip install gym==0.21.0 but got the cryptic error: Collecting gym==0.21.0 Using cached gym-0.21.0.tar.gz (1.5 MB) Preparing metadata (setup.py) ... error error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [1 lines of output] error in gym setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details.  What may be causing this error?

Answer: Based on the comments in the issue section at https://github.com/openai, these are the changes that need to be made -  pip install setuptools==65.5.0 pip==21 # gym 0.21 installation is broken with more recent versions pip install wheel==0.38.0  I was facing the same issue and after doing the above, it was resolved. The problem has been addressed here and here.
Question: I have simple Config class from FastAPI tutorial. But it seems like it uses old pydantic version. I run my code with pydantic v2 version and get a several errors. I fix almost all of them, but the last one I cannot fix yet. This is part of code which does not work: from pydantic import AnyHttpUrl, HttpUrl, PostgresDsn, field_validator from pydantic_settings import BaseSettings from pydantic_core.core_schema import FieldValidationInfo load_dotenv() class Settings(BaseSettings): ... POSTGRES_SERVER: str = 'localhost:5432' POSTGRES_USER: str = os.getenv('POSTGRES_USER') POSTGRES_PASSWORD: str = os.getenv('POSTGRES_PASSWORD') POSTGRES_DB: str = os.getenv('POSTGRES_DB') SQLALCHEMY_DATABASE_URI: Optional[PostgresDsn] = None @field_validator("SQLALCHEMY_DATABASE_URI", mode='before') @classmethod def assemble_db_connection(cls, v: Optional[str], info: FieldValidationInfo) -> Any: if isinstance(v, str): return v postgres_dsn = PostgresDsn.build( scheme="postgresql", username=info.data.get("POSTGRES_USER"), password=info.data.get("POSTGRES_PASSWORD"), host=info.data.get("POSTGRES_SERVER"), path=f"{info.data.get('POSTGRES_DB') or ''}", ) return str(postgres_dsn)  That is the error which I get: sqlalchemy.exc.ArgumentError: Expected string or URL object, got MultiHostUrl('postgresql://user:password@localhost:5432/database')  I check a lot of places, but cannot find how I can fix that, it looks like build method pass data to the sqlalchemy create_engine method as a MultiHostUrl instance instead of string. How should I properly migrate this code to use pydantic v2? UPDATE I have fixed that issue by changing typing for SQLALCHEMY_DATABASE_URI: Optional[PostgresDsn] = None to SQLALCHEMY_DATABASE_URI: Optional[str] = None. Because pydantic makes auto conversion of result for some reason. But I am not sure if that approach is the right one, maybe there are better way to do that?

Answer: You can use unicode_string() to stringify your URI as follow: from sqlalchemy.ext.asyncio import create_async_engine create_async_engine(settings.POSTGRES_URI.unicode_string())  Check documentation page here for additional explanation.
Question: I wrote a script that converts currencies based on the current exchange rate. It seems to work fine, except when trying to convert EUR to USD, as it never gets the exchange rate correctly. For example, it tells me 1000 EUR are worth 64 USD, whereas the reality would be around 950...The actual exchange rate is 1.06 but forex mistakes it for 0.06 How can I fix it? Here is my code: Module 1: from forex_python.converter import CurrencyRates def exchange_rate(c1,c2, time): c = CurrencyRates() return c.get_rate(c1,c2, time)  Module 2: from currency_rates import exchange_rate from datetime import datetime # List of available currencies list_of_currencies = ["USD", "EUR", "GBP", "ILS", "DKK", "CAD", "IDR", "BGN", "JPY", "HUF", "RON", "MYR", "SEK", "SGD", "HKD", "AUD", "CHF", "KRW", "CNY", "TRY", "HRK", "NZD", "THB", "LTL", "NOK", "RUB", "INR", "MXN", "CZK", "BRL", "PLN", "PHP", "ZAR"] # Get user input for currency 1, amount and currency 2 c1 = input(f"""Which currency would you like to convert from?\n {list_of_currencies}\n""") if c1 not in list_of_currencies: print("Invalid answer.") value_c1 = input("How much of that currency?\n") c2 = input(f"""Which currency would you like to convert to? \n {list_of_currencies}\n""") if c2 not in list_of_currencies: print("Invalid answer.") # Calculate result based on exchange rate # Get current time now = datetime.now() rate = exchange_rate(c1,c2, now) result = float(rate) * float(value_c1) # Print result print(f"{value_c1} {c1} are worth {result} {c2}.")  Something needs to be done with the rate itself. But adding 1 when it's below 1 is not a solution as some other exchange rates might be below 1 for real.

Answer: There is no issue with your code or the forex_python.converter library. This is a bug with the :latest version of theforexapi which is used by the forex_python.converter, see link to raised issue: latest/?base=EUR&symbols=USD returns base=ILS #18  The returned "base" is "ILS" instead of "EUR" as expected. This bug only occurs for latest/?base=EUR&symbols=USD. If you specify a date instead of latest, or if you use any other combination of base and symbols, the bug does not happen.
Question: I have some data from the lab, which I'm fitting to the function f(s, params) = amp * (s/sp) / (1 + s/sp + 1.041) + bg (couldn't figure out how to type set this) I set absolute_sigma=True on curve_fit to get the absolute uncertainties of the parameters (sp, amp, bg), but the calculated error from np.sqrt(np.diag(pcov)) seems unreasonable. See the plot below. The blue dots are data. The red line is the f(s, *popt). The green line replaces the optimal sp value with sp - it's error as calculated from np.sqrt(np.diag(pcov)). The orange line is sp + the same value.  I would expect the +/- lines to be much tighter to the red line. Here's a minimal example: # Function for fitting def scattering_rate(s, sp, amp, bg): return amp * (s/sp) / (1 + s/sp + (-20/19.6)**2) + bg # data s = np.array([0.6, 1.2, 2.3, 4.3, 8.1, 15.2, 28.5, 53.4]) y = np.array([8.6, 8.5, 8.9, 9.5, 10.6, 12.6, 15.5, 18.3]) # Fit data to saturated scattering rate popt, pcov = curve_fit(scattering_rate, s, y, absolute_sigma=True) print('Fit parameters', popt) print('Fit uncertainties', np.sqrt(np.diag(pcov))) # Construct fit from optimized parameters fit_s = np.linspace(np.min(s), np.max(s), 100) fit = scattering_rate(fit_s, *popt) # Consider one error difference in sp value fit_plus_err = saturation_power(fit_s, popt[0] + np.sqrt(np.diag(pcov))[0], popt[1], popt[2]) fit_minus_err = saturation_power(fit_s, popt[0] - np.sqrt(np.diag(pcov))[0], popt[1], popt[2]) # Plot plt.plot(s, y, '-o', markeredgecolor='k', label='data') plt.plot(fit_s, fit_plus_err, label='sp + err') plt.plot(fit_s, fit_minus_err, label='sp - err') plt.plot(fit_s, fit, '-r', label='opt sp') plt.xlabel('s') plt.ylabel('y') plt.legend() plt.grid() plt.show()  Edit Following @jlandercy's answer, we need the error bars of the original data which are y_err = array([0.242, 0.231, 0.282, 0.31 , 0.373]). Including that in curve_fit's sigma argument, the results look much better though still a bit distance 

Answer: Why When using absolute_sigma=True your are advised to feed the sigma switch as well, if not they are replaced by 1. making Chi Square loss function behaves as RSS and pcov becomes somehow meaningless if your uncertainty are not unitary. Fix Provides uncertainty or an estimate of it as well to get a meaningful pcov matrix. Setting sigma to 0.15 with your data drastically improve the pcov matrix while keeping the Chi Square statistics significant: sy = np.full(y.size, 0.15) popt, pcov = curve_fit(scattering_rate, s, y, sigma=sy, absolute_sigma=True) np.sqrt(pcov) # array([[3.25479068, 2.07084837, 0.45391942], # [2.07084837, 1.35773667, 0.2563505 ], # [0.45391942, 0.2563505 , 0.09865549]])    Update You have two direct options to improve the error:  Collect more points (5 points for 3 parameters leave a Chi Square with dof=2 which is very asymmetric); Reduce the uncertainty on y.  Figure below shows the impact on a synthetic dataset with equivalent parameters of yours:
Question: I have a table with a JSONB field and would like to insert into it using a named dict like so: sql = "INSERT INTO tbl (id, json_fld) VALUES (%(id)s, %(json_fld)s)" conn.execute(sql, {'id':1, 'json_fld': {'a':1,'b':false, 'c': 'yes'}});  I tried the answers in this question but those all apply to psycopg2 and NOT psycopg3 and they do not work here (notably I tried): conn.execute(sql, {'id':1, 'json_fld': json.dumps({'a':1,'b':false, 'c': 'yes'})});  The error remains the same:  psycopg.ProgrammingError: cannot adapt type 'dict' using placeholder '%s' (format: AUTO) 

Answer: Python code to convert dict to jsonb using psycopg JSON adapters described here JSON adaptation section JSON adaptation. import psycopg from psycopg.types.json import Jsonb con = psycopg.connect("dbname=test user=postgres") cur = con.cursor() cur.execute("insert into json_test values(%s, %s)", [1, Jsonb({'a':1,'b': False, 'c': 'yes'})]) con.commit()  This results in: select * from json_test ; id | js_fld ----+---------------------------------- 1 | {"a": 1, "b": false, "c": "yes"}
Question: I'd like to blur the edges of a surface in a "gradient fashion", in Pygame. Here is an example of the desired effect where we have :  No blur on first image Medium blur on second square High blur on third square  image of blur effects desired Pygame has 2 functions to blur surfaces : pygame.transform.box_blur and pygame.transform.gaussian_blur. However they don't have perform a gradient blur between the surface and the screen like in the above example.

Answer: You need to create a Surface with a transparent background (pygame.SRCALPHA) that is slightly larger than your image. Then you can blit your image onto the surface and blur it: rect = pygame.Rect(0, 0, image.get_width()+100, image.get_height()+100) blur_image = pygame.Surface(rect.size, pygame.SRCALPHA) blur_image.blit(image, image.get_rect(center = rect.center))  If you then blit this Surface onto the display, you get a gradient transition. When you look in the pygame documentation you'll find, that pygame.transform.box_blur and pygame.transform.gaussian_blur does not exist ins Pygame (2.5.1 at the time of writing) but only pygame-ce. However, you can create your own blur function with PLI Image: def blur_image(surf, radius): pil_string_image = pygame.image.tostring(surf, "RGBA",False) pil_image = Image.frombuffer("RGBA", surf.get_size(), pil_string_image) pil_blurred = pil_image.filter(ImageFilter.GaussianBlur(radius=radius)) blurred_image = pygame.image.fromstring(pil_blurred.tobytes(), pil_blurred.size, pil_blurred.mode) return blurred_image.convert_alpha()   Minimal example  import pygame from PIL import Image, ImageFilter pygame.init() window = pygame.display.set_mode((300, 300)) clock = pygame.time.Clock() background = pygame.Surface(window.get_size()) ts, w, h, c1, c2 = 50, *background.get_size(), (32, 32, 32), (64, 64, 64) tiles = [((x*ts, y*ts, ts, ts), c1 if (x+y) % 2 == 0 else c2) for x in range((w+ts-1)//ts) for y in range((h+ts-1)//ts)] [pygame.draw.rect(background, color, rect) for rect, color in tiles] def blur_image(surf, radius): pil_string_image = pygame.image.tostring(surf, "RGBA",False) pil_image = Image.frombuffer("RGBA", surf.get_size(), pil_string_image) pil_blurred = pil_image.filter(ImageFilter.GaussianBlur(radius=radius)) blurred_image = pygame.image.fromstring(pil_blurred.tobytes(), pil_blurred.size, pil_blurred.mode) return blurred_image.convert_alpha() def blur_image_edges(surf, radius): rect = pygame.Rect(0, 0, surf.get_width()+radius*4, surf.get_height()+radius*4) blur_surf = pygame.Surface(rect.size, pygame.SRCALPHA) blur_surf.blit(surf, surf.get_rect(center = rect.center)) blurred_image = blur_image(blur_surf, radius) return blurred_image image = pygame.Surface((200, 200)) image.fill('red') blurred_image = blur_image_edges(image, 20) run = True while run: clock.tick(100) for event in pygame.event.get(): if event.type == pygame.QUIT: run = False window.blit(background, (0, 0)) window.blit(blurred_image, blurred_image.get_rect(center = window.get_rect().center)) pygame.display.flip() pygame.quit() exit()
Question: I'm working with a rasterio raster that I've 'read' into python, so it is now a numpy array. The outer edge of the np array are all zeros, and the interior are all ones, except, in the midst of the ones are occasional zeros, see the example array below. I want to leave all the zeros on the outside of the array alone (i.e. keep them zeros), but want to convert the zeros that are completely surrounded by ones (i.e. the zeros in the middle of the donut of ones) to one. However, I'm not really sure how to start. Current array: import numpy as np arr = np.array([[0, 0, 0, 0, 0, 0, 0, 0], [0, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 0, 0, 1, 1, 0], [0, 1, 1, 0, 1, 1, 0, 0], [0, 1, 1, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]])  Goal array: import numpy as np arr = np.array([[0, 0, 0, 0, 0, 0, 0, 0], [0, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 1, 1, 1, 0, 0], [0, 1, 1, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]]) 

Answer: Perhaps you can use scipy.ndimage.binary_fill_holes: import numpy as np from scipy.ndimage import binary_fill_holes arr = np.array([[0, 0, 0, 0, 0, 0, 0, 0], [0, 1, 1, 1, 1, 1, 1, 0], [0, 1, 1, 0, 0, 1, 1, 0], [0, 1, 1, 0, 1, 1, 0, 0], [0, 1, 1, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]]) filled_arr = binary_fill_holes(arr).astype(int) print(filled_arr)  Output: [[0 0 0 0 0 0 0 0] [0 1 1 1 1 1 1 0] [0 1 1 1 1 1 1 0] [0 1 1 1 1 1 0 0] [0 1 1 1 1 1 0 0] [0 0 0 0 0 0 0 0]]
Question: How do I make it so that it shows all x ticks from 0 to 9?  bin diff 1 4 -0.032748 3 9 0.106409 13 7 0.057214 17 3 0.157840 19 0 -0.086567 ... ... ... 1941 0 0.014386 1945 4 0.049601 1947 9 0.059406 1957 1 0.045282 1959 6 -0.033853   ( so.Plot(x='bin', y='diff', data=diff_df) .theme({**axes_style("whitegrid"), "grid.linestyle": ":"}) .add(so.Dots()) .add(so.Range(color='orange'), so.Est()) .add(so.Dot(color='orange'), so.Agg()) .add(so.Line(color='orange'), so.Agg()) .label( x="Image Similarity Bin", y="Difference", color=str.capitalize, ) )  I tried to set xticks in .label, but it doesn't do anything. 

Answer:  As per Parameterizing scales and Customizing legends and ticks, use seaborn.objects.Continuous.tick inside seaborn.objects.Plot.scale  .scale(x=so.Continuous().tick(every=1))    import pandas as pd import numpy as np import seaborn.objects as so # sample data np.random.seed(365) rows = 1100 data = {'diff': np.random.random(size=rows), 'bin': np.random.choice(range(10), size=rows)} df = pd.DataFrame(data) # plot (so.Plot(x='bin', y='diff', data=df) .add(so.Dots()) .add(so.Range(color='orange'), so.Est()) .add(so.Dot(color='orange'), so.Agg()) .add(so.Line(color='orange'), so.Agg()) .label(x="Image Similarity Bin", y="Difference", color=str.capitalize) .scale(x=so.Continuous().tick(every=1))) # adjust the ticks here
Question: I'm using Poetry to manage my Python project, and I would like to check if a package is installed. This means the package is actually fetched from the remote repository and is located in the .venv/ folder. I went through the official documentation but didn't find any command that can do that. Any ideas? Thanks. Update: I ended up with a solution by running the following command and parsing its output. Thanks all for the help here! poetry install --dry-run --sync --no-ansi

Answer: Assuming you have activated the virtual environment in the .venv folder (using, for example source .venv/bin/activate), you can use pip list to list all of the installed packages in that virtual environment. There is a poetry show command to list all available packages as well that you might want to look into: https://python-poetry.org/docs/cli/#show Update: You can parse the output of the poetry install --dry-run command to see what packages are installed, without actually installing any packages. Look for the lines with both "Installing" and "Skipped" and "reason: Already installed", to see what packages are already installed.
Question: import polars as pl df = pl.DataFrame(['2023-08-01T06:13:24.448409', '2023-08-01T07:29:34.491027']) print(df.with_columns(pl.col('column_0').str.strptime(pl.Date,'%Y-%m-%d')))  So I have the following mass of code, and for some ungodly reason I cant for the death of me extract the date from those given strings. In the following example I keep getting the error exceptions.ComputeError: strict date parsing failed for 2 value(s) (2 unique): ["2023-08-01T06:13:24.448409", "2023-08-01T07:29:34.491027"]  Any ideas on how can I extract the string from such format of datetime? And why I get this error?

Answer: To parse a datetime upfront you'll need to provide the full format (date & time) initially then retrieve the date part afterwards. Try the following format: '%Y-%m-%dT%H:%M:%S.%f'
Question: Setting the value of a new dataframe column: df.loc[df["Measure] == metric.label, "source_data_url"] = metric.source_data_url  now (as of Pandas version 2.1.0) gives a warning, FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value ' metric_3' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.  The Pandas documentation discusses how the problem can be solved for a Series but it is not clear how to do this iteratively (the line above is called in a loop over metrics and it's the final metric that gives the warning) when assigning a new DataFrame column. How can this be done?

Answer: I had the same problem. My intuition of this is that when you are setting value for the first time to the column source_data_url, the column does not yet exists, so pandas creates a column source_data_url and assigns value NaN to all of its elements. This makes Pandas think that the column's dtype is float64. Then it raises this warning. My solution was to create the column with some default value, e.g. empty string, before adding values to it: df["source_data_url"] = "" or None seems also to work: df["source_data_url"] = None