eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a pandas dataframe which looks like the following:  site pay delta over under phase a a b ID D01 London 12.3 10.3 -2.0 0.0 -2.0 D02 Bristol 7.3 13.2 5.9 5.9 0.0 D03 Bristol 17.3 19.2 1.9 1.9 0.0  I'd like to flatten the column multindex to the columns are ID site a b delta over under D01 London 12.3 10.3 -2.0 0.0 -2.0 D02 Bristol 7.3 13.2 5.9 5.9 0.0 D03 Bristol 17.3 19.2 1.9 1.9 0.0  I'm struggling with the online documentation and tutorials to work out how to do this. I'd welcome advice, ideally to do this in a robust way which doesn't hardcode column positions.  UPDATE: the to_dict is {'index': ['D01', 'D02', 'D03'], 'columns': [('site', 'a'), ('pay', 'a'), ('pay', 'b'), ('delta', ''), ('over', ''), ('under', '')], 'data': [['London', 12.3, 10.3, -2.0, 0.0, -2.0], ['Bristol', 7.3, 13.2, 5.8999999999999995, 5.8999999999999995, 0.0], ['Bristol', 17.3, 19.2, 1.8999999999999986, 1.8999999999999986, 0.0]], 'index_names': ['ID'], 'column_names': [None, 'phase']} 

Answer: As the above answer mentioned the requirement is not clear, I have tried out just the flattening of the data, let us know import pandas as pd data = { 'index': ['D01', 'D02', 'D03'], 'columns': [('site', 'a'), ('pay', 'a'), ('pay', 'b'), ('delta', ''), ('over', ''), ('under', '')], 'data': [['London', 12.3, 10.3, -2.0, 0.0, -2.0], ['Bristol', 7.3, 13.2, 5.9, 5.9, 0.0], ['Bristol', 17.3, 19.2, 1.9, 1.9, 0.0]], 'index_names': ['ID'], 'column_names': [None, 'phase'] } # Construct the MultiIndex columns and DataFrame multiindex_columns = pd.MultiIndex.from_tuples(data['columns'], names=data['column_names']) df = pd.DataFrame(data['data'], index=data['index'], columns=multiindex_columns) # Flattening the MultiIndex columns df.columns = [' '.join(filter(None, col)) for col in df.columns] print(df)  Flattened DataFrame:  site a pay a pay b delta over under D01 London 12.30 10.30 -2.00 0.00 -2.00 D02 Bristol 7.30 13.20 5.90 5.90 0.00 D03 Bristol 17.30 19.20 1.90 1.90 0.00
Question: I am doing some statistic of a very large dataframe that takes sums of multiple random samples. I would like the samples to be disjuct (no number should be present in two different samples). Minimal example that might use some numbers multiple times: import polars as pl import numpy as np df = pl.DataFrame( {"a": np.random.random(1000)} ) N_samples = 50 N_logs = 20 sums = [ df.sample(N_logs).select(pl.col("a").sum()).item() for _ in range(N_samples) ]  How to avoid multiple usage of same numbers?

Answer: You can sample them all at once using with_replacement = False (which is default) and then aggregate into N_samples sums: ( df .sample(N_samples * N_logs) .group_by(pl.int_range(pl.len()) // N_logs) .sum() .get_column("a") )  shape: (50,) Series: 'a' [f64] [ 9.993712 10.667377 9.983055 7.092786 10.780031 … 9.384218 8.57084 10.085927 12.77378 10.23612 ]
Question: In Django 5.1 {% querystring %} was added. Is there some way to use it with GET form? For example, let's say we have template with: <span>Paginate by:</span> <a href="{% querystring paginate_by=50 %}">50</a> {# ... #} <form method="GET"> <input name="query" value="{{ request.GET.query }}"> <button type="submit">Search</button> </form>  Assuming that we are currently on localhost:8000/?paginate_by=50, how to change form so clicking Search won't delete paginate_by query parameter - so what I want is for example localhost:8000/?paginate_by=50&query=abc and not localhost:8000/?query=abc? Before 5.1 I handled that via providing form with hidden fields based on GET parameters, but I am hoping that now more elegant solution is possible.

Answer:  Is there some way to use it with GET form?  No., the {% querystring … %} template tag [Django-doc] is used to generate querystrings. This is not what a form element is supposed to do. If you add a querystring at the end of the action=".." [mdn-doc] for a form that makes a GET request, then the browser will normally strip away the querystring. You probably can make use of the template tag, and some JavaScript that thus when submitting the form obtains the data from the form, and from the {% querystring %} (or probably better, JavaScript's window.location.search), and mix these together. Then you thus prevent the default form flow, but it thus requires some JavaScript to handle it. I would however advise not to manually render HTML forms if you are searching. You can use django-filter [GitHub] to automatically generate a filter form, that will thus preserve other options.
Question: Suppose my data look as follows: import datetime import pandas as pd df = pd.DataFrame({'datetime': [datetime.datetime(2024, 11, 27, 0), datetime.datetime(2024, 11, 27, 1), datetime.datetime(2024, 11, 28, 0), datetime.datetime(2024, 11, 28, 1), datetime.datetime(2024, 11, 28, 2)], 'product': ['Apple', 'Banana', 'Banana', 'Apple', 'Banana']}) datetime product 0 2024-11-27 00:00:00 Apple 1 2024-11-27 01:00:00 Banana 2 2024-11-28 00:00:00 Banana 3 2024-11-28 01:00:00 Apple 4 2024-11-28 02:00:00 Banana  All I want is to plot the relative frequencies of the products sold at each day. In this example 1/2 (50%) of apples and 1/2 of bananas on day 2024-11-27. And 1/3 apples and 2/3 bananas on day 2024-11-28  What I managed to do: absolute_frequencies = df.groupby([pd.Grouper(key='datetime', freq='D'), 'product']).size().reset_index(name='count') total_counts = absolute_frequencies.groupby('datetime')['count'].transform('sum') absolute_frequencies['relative_frequency'] = absolute_frequencies['count'] / total_counts absolute_frequencies.pivot(index='datetime', columns='product', values='relative_frequency').plot()  I am pretty confident, there is a much less complicated way, since for the absolute frequencies I simply can use: df.groupby([pd.Grouper(key='datetime', freq='D'), 'product']).size().unstack('product').plot(kind='line') 

Answer: You can use a crosstab with normalize: ct = pd.crosstab(df['datetime'].dt.normalize(), df['product'], normalize='index')  Output: product Apple Banana datetime 2024-11-27 0.500000 0.500000 2024-11-28 0.333333 0.666667  As a graph: ct.plot.bar()  Output:
Question: I have a data structure like this: class Pizza(models.Model): name = models.CharField(max_length=100) toppings = models.ManyToManyField(Topping, related_name="pizzas") class Topping(models.Model): name = models.CharField(max_length=100)  And to get all topping IDs related to a pizza I can do this: list(map(lambda t: t.id, pizza.toppings.all()))  But this fetches all toppings completely from the database, even thought I only need the IDs. Is there a way to get the IDs without fetching the complete objects (for performance reasons)?

Answer: Use .values_list() methods to get only the IDs as tuples: topping_ids = pizza.toppings.all().values_list("id") # Example of result ( (4), (5), (12), (54) )
Question: I've tried numerous times to register an app and connect to in in python: app_id = '670...' tenant_id = '065...' client_secret_value = 'YJr...' import requests import msal authority = f'https://login.microsoftonline.com/{tenant_id}' scopes = ['https://analysis.microsoft.net/powerbi/api/.default'] app = msal.ConfidentialClientApplication(app_id, authority=authority, client_credential=client_secret_value) result = None result = app.acquire_token_for_client(scopes=scopes)  Overview:   I feel like I've followed this video exactly: https://www.youtube.com/watch?v=3Fu8FjvYvyc&t=577s&ab_channel=JJPowerBI I'm up to minute 8:38. I'm getting the following error messsage and googling it shows me the tenatid should be the id and not the domain name. I'm not sure why that's happening and what I need to change to get this to work  Edit adding API Permissions I am the owner of the subscription.    Edit2: Looks a little different then the comment, but I enabled this and it says it could take 15 minutes to update. 

Answer: The error occurred as you are using wrong scope value to generate access token for Power BI API. Initially, I too got same error when I tried to generate token with scope as https://analysis.microsoft.net/powerbi/api/.default like this:  Make sure to use https://analysis.windows.net/powerbi/api/.default/ as scope value that worked and generated token successfully as below: app_id = 'appId' tenant_id = 'tenantId' client_secret_value = 'secret' import msal authority = f'https://login.microsoftonline.com/{tenant_id}' scopes = ['https://analysis.windows.net/powerbi/api/.default'] app = msal.ConfidentialClientApplication( app_id, authority=authority, client_credential=client_secret_value ) result = app.acquire_token_for_client(scopes=scopes) if 'access_token' in result: print(result['access_token']) else: print(result)  Response:
Question: I am trying to wrap a C pointer array of type size_t with a numpy ndarray via Cython using the following: cimport numpy as cnp from libcpp.vector cimport vector cnp.import_array() cdef size_t num_layers = 10 cdef vector[size_t] steps_taken_vec = vector[size_t]() steps_taken_vec.resize(3 * num_layers) cdef size_t* steps_taken_ptr = steps_taken_vec.data() cdef cnp.npy_intp[2] shape = [3, num_layers] cdef cnp.npy_intp ndim = 2 self.shooting_method_steps_taken_array = cnp.PyArray_SimpleNewFromData( ndim, &shape[0], cnp.NPY_UINTP, # <-- This is the problem steps_taken_ptr) # steps_taken_ptr is a size_t*  The above produces the error "cimported module has no attribute 'NPY_UINTP'". According to numpy's documentation there should be an enum that directs numpy to create an array using size_t: https://numpy.org/devdocs/reference/c-api/dtype.html#c.NPY_TYPES.NPY_UINTP The PyArray_SimpleNewFromData API requires a enum that defines the type used to create the ndarray. However, the actual init.pxd does not appear to have that enum. It does set the type correctly, see line 25 here but no enum in this list. Those links, and my code, is using numpy 1.26.4. I looked ahead at 2.0+ and see that there was some definitional changes to this type, but the enum still appears to be missing (see here) As a workaround, I am using cnp.NPY_UINT64 which works but I am not sure if that is guranteed to be the same size as size_t across platforms and into the future. Am I missing something here?

Answer: The easiest solution is probably to declare NPY_UINTP yourself. def extern from *: cdef enum: NPY_UINTP  This just assumes Cython that some enum-like constant exists called NPY_UINTP. You can pass that to PyArray_SimpleNewFromData. It doesn't matter that it's not from the "official" Numpy pxd file.  As I said in a comment, do report the missing constant to Numpy (or better yet, contribute a fix to Numpy). If they don't know then they can't fix it.
Question: I have a tensor a that is of shape (n/f, c, c) that I want to multiply by another tensor b of shape (n, c, 1). Each row of a represents f rows of b, such that the naiive way of implementing this would be to simply repeat each row of a f times before performing the multiplication: n = 100 c = 5 f = 10 a = tf.constant(np.random.rand(n//f, c, c)) b = tf.constant(np.random.rand(n, c, c)) a_prime = tf.repeat(a, f, 0) result = a_prime @ b  This works, but for large n and f I'm worried about the memory footprint of the repeat. I could of course loop through each row and perform dot-products manually, but that would have implications on performance. Is there a better way?

Answer: We can do this by reshaping tensors and utilizing broadcasting, We can perform matrix multiplication more efficiently by eliminating the need for explicit repetition. import tensorflow as tf import numpy as np n = 100 c = 5 f = 10 a = tf.constant(np.random.rand(n // f, c, c)) b = tf.constant(np.random.rand(n, c, c)) #Reshape a and b a_reshaped = tf.reshape(a, (1, n // f, c, c)) b_reshaped = tf.reshape(b, (n, 1, c, c)) # perform matrix multiplication result = tf.matmul(a_reshaped, b_reshaped) result = tf.reduce_sum(result, axis=1) print(result.shape)  output: (100, 5, 5)
Question: I'm trying to deploy a simple app (learning log from Python Crash Course) to heroku. The app runs but upon login, I get a 500 error, and in debug=true the error is: no such table: auth_user. I realise this has something to do with squlite3 and Postgresql, but when I try to build the code in settings: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'NAME': BASE_DIR / 'db.sqlite3', }  } the error message is ImportError: DLL load failed while importing _psycopg: The specified module could not be found.  I have attempted to install and import psycopg2, and it appears to be in requirements.txt. But I think the paths are not aligning, since psycopg2 is in a python pathway, and setting.py is looking in my project environment. I'm very confused! Please help!

Answer: You have not correctly configured your DATABASES in settings.py. You are attempting to use the default sqlite3 configuration with a PostgreSQL engine which wouldn't work. Also Heroku does not work well with sqlite3 and it wouldn't be a good idea to use it in production. See Why are my file uploads missing/deleted from the Application?. It's great you've installed psycopg2 - the database adapter for PostgreSQL. You'd usually have to configure your PostgreSQL database as shown below: DATABASES = { "default": { "ENGINE": "django.db.backends.postgresql", "NAME": "mydatabase", "USER": "mydatabaseuser", "PASSWORD": "mypassword", "HOST": "127.0.0.1", "PORT": "5432", } }  Ensure you've created your database and enter the credentials in the DATABASES dictionary in settings.py. When deploying to Heroku, you'd have to provision your database on Heroku by selecting the Postgre addon. Please see Deploying Django App to Heroku: Full Guide for a complete guide to deploying your Django app on Heroku.
Question: How do I interpret the Pressure values in this dataset? I downloaded ERA5 Potential Vorticity and Pressure(Monthly means of Daily means, analysis, Potential Temperature levels). The pressure data doesn’t make sense to me. Please see the attached image. The potential vorticity data makes sense. It is a 2D grid with the same shape as the lats/lons. But the pressure data is 1D with a length that doesn’t match anything I can think of. Any advice? Thanks!  Download request: #!/usr/bin/env python import cdsapi c = cdsapi.Client() c.retrieve("reanalysis-era5-complete", { "class": "ea", "date": "20200101/20200201/20200301/20200401/20200501/20200601/20200701/20200801/20200901/20201001/20201101/20201201/20210101/20210201/20210301/20210401/20210501/20210601/20210701/20210801/20210901/20211001/20211101/20211201/20220101/20220201/20220301/20220401/20220501/20220601/20220701/20220801/20220901/20221001/20221101/20221201/20230101/20230201/20230301/20230401/20230501/20230601/20230701/20230801/20230901/20231001/20231101/20231201/20240101/20240201/20240301/20240401/20240501/20240601/20240701/20240801", "expver": "1", "levelist": "265/275/285/300/315/320/330/350/370/395/430/475/530/600/700/850", "levtype": "pt", "param": "54.128/60.128", "stream": "moda", "type": "an" }, "output") 

Answer: So if you convert your grib file to netcdf like this cdo -f nc4 copy test.grb test.nc4  and look at the header with ncdump ncdump -h test.nc4  you will see that the PV field (var60) is retrieved on an unstructured reduced Gaussian grid and the pressure field (var54) is a spectral field, that's why you have the zeros as these are the spectral coefficients. float var54(time, lev, nsp, nc2) ; var54:table = 128 ; var54:CDI_grid_type = "spectral" ; var54:axis = "TZ--" ; var54:truncation = 639 ; float var60(time, lev, rgrid) ; var60:table = 128 ; var60:CDI_grid_type = "gaussian_reduced" ; var60:CDI_grid_num_LPE = 320 ; var60:CDI_grid_latitudes = "lat" ; var60:CDI_grid_reduced_points = "reduced_points" ;  This makes conversion to a regular grid problematic with cdo, you can do this with eccodes though. However, a far easier solution is to simply request a regular lat-lon grid in the original retrieval, and let MARS handle the interpolation for you. You can do this with the keyword "grid", like this (I also use the format keyword to get the file directly as netcdf, but of course you can stay with grib if you prefer). #!/usr/bin/env python import cdsapi c = cdsapi.Client() c.retrieve("reanalysis-era5-complete", { "class": "ea", "date": "20200101", "expver": "1", "levelist": "265/275/285/300/315/320/330/350/370/395/430/475/530/600/700/850", "levtype": "pt", "param": "54.128/60.128", "stream": "moda", "type": "an", "grid": "F128", "format":"netcdf" }, "test2.nc")  This retrieves the data directly on a regular lat-lon grid. My test2.nc file now has these dimensions: longitude = 512 ; latitude = 256 ; theta = 16 ;  And here are my regular-gridded field headers for the two variables: float pres(theta, latitude, longitude) float pv(theta, latitude, longitude)  I used a low resolution 128 grid here, but you can select others. See the WIKI for the grid options.
Question: I'm trying to create an image captioning model using hugging face blip2 model on colab. My code was working fine till last week (Nov 8) but it gives me an exception now. To install packages I use the following command: !pip install -q git+https://github.com/huggingface/peft.git transformers bitsandbytes datasets  To load blip2 processor and model I use the following code: model_name = "Salesforce/blip2-opt-2.7b" processor = AutoProcessor.from_pretrained(model_name) model = Blip2ForConditionalGeneration.from_pretrained(model_name,device_map="auto",load_in_8bit=False)  I use the following code to generate captions: def generate_caption(processor, model, image_path): image = PILImage.open(image_path).convert("RGB") print("image shape:" + image.size) device = "cuda" if torch.cuda.is_available() else "cpu" # Preprocess the image inputs = processor(images=image, return_tensors="pt").to(device) print("Input shape:", inputs['pixel_values'].shape) print("Device:", device) # Additional debugging for key, value in inputs.items(): print(f"Key: {key}, Shape: {value.shape}") # Generate caption with torch.no_grad(): generated_ids = model.generate(**inputs) caption = processor.decode(generated_ids[0], skip_special_tokens=True) return caption  here is the code that uses this method to generate captions:  image_path = "my_image_path.jpg" caption = generate_caption(processor, model, image_path) print(f"{image_path}: {caption}"  finally, this is the outputs and errors of running the code above: image shape: (320, 240) Input shape: torch.Size([1, 3, 224, 224]) Device: cuda Key: pixel_values, Shape: torch.Size([1, 3, 224, 224]) --------------------------------------------------------------------------- . . . /usr/local/lib/python3.10/dist-packages/transformers/models/blip_2/modeling_blip_2.py in generate(self, pixel_values, input_ids, attention_mask, interpolate_pos_encoding, **generate_kwargs) 2314 if getattr(self.config, "image_token_index", None) is not None: 2315 special_image_mask = (input_ids == self.config.image_token_index).unsqueeze(-1).expand_as(inputs_embeds) -> 2316 inputs_embeds[special_image_mask] = language_model_inputs.flatten() 2317 else: 2318 logger.warning_once( RuntimeError: shape mismatch: value tensor of shape [81920] cannot be broadcast to indexing result of shape [0]  I have searched the internet and used various AI models for help but to no avail. My guess is that this is a package update problem since my code had no problem last week. (I tried to restore my code to Nov 8 version but it throws an exception.) Moreover, I don't understand how 81920 is calculated in the error message.

Answer: I had the same issue. You need to add a prompt in the processor: prompt = " " inputs = processor(images=image, text=prompt, return_tensors="pt").to(device="cuda", dtype=torch.float16)  Hope it helps.
Question: I am using SQLAlchemy and PostgreSQL, and I am trying to create a case-insensitive unique constraint that works like this  UniqueConstraint( 'country', 'organisation_id', func.lower(func.trim('name')), name='uq_country_orgid_lower_trim_name' )  Ensuring a unique combination of name, country and organisation id, regardless of case and spacing in the name, i.e. "Name 1", "name1", "nAmE 1" would all be handled as "name1" in the check. I want to make sure that I do not change the actual case or spacing of the name saved in the database. How would I go about this?

Answer: This is mostly the ORM version of @snakecharmerb and avoids the use of text (nothing wrong with sanitized text, it is just a preference). This answer suggests using a unique index, instead of unique constraint because SQLA lacks support. from sqlalchemy import create_engine, func, Index, select, column from sqlalchemy.exc import IntegrityError from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, Session class Base(DeclarativeBase): ... class SomeTable(Base): __tablename__ = "some_table" __table_args__ = ( Index( "uq_country_orgid_lower_trim_name", "country", "organization_id", func.lower(func.replace(column("name"), " ", "")), unique=True, ), ) name: Mapped[str] id: Mapped[int] = mapped_column(primary_key=True) country: Mapped[str] organization_id: Mapped[int] engine = create_engine("postgresql+psycopg://") Base.metadata.create_all(engine) with Session(engine) as session: session.add(SomeTable(country="C1", organization_id=1, name="NAME 1")) session.commit() with Session(engine) as session: rows = ( SomeTable(country="C1", organization_id=1, name="NAME1"), SomeTable(country="C1", organization_id=1, name="Name 1"), SomeTable(country="C1", organization_id=1, name="name1"), SomeTable(country="C1", organization_id=1, name="nAmE 1"), ) for row in rows: try: with session.begin_nested(): session.add(row) except IntegrityError: print("failed as expected") with Session(engine) as session: row = session.scalar(select(SomeTable)) assert row assert row.name == "NAME 1"
Question: I am playing with GroupedData in pyspark. This is my environment. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.5.1 /_/ Using Scala version 2.12.18, OpenJDK 64-Bit Server VM, 11.0.24 Branch HEAD  https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.GroupedData.html I wonder if the following is possible. Say I want to use only the methods of GroupedData, and not import any functions from pyspark.sql.functions. OK, suppose I have some DataFrame and I've grouped it already by column A and I've got a GroupedData object back. Now I want to do on my GroupedData object say sum(column B), and say avg(column C) and maybe min(column D) in one shot or via chained method calls. Can I do this just by using GroupedData methods? I am asking this because it seems that once I've done sum(column B), I don't have a GroupedData object anymore, and so I cannot continue to chain any GroupedData methods further. So is that (what I have in mind) possible or not? If it's possible, how can we do it?

Answer: I do not think that this is possible. Looking at the source of GroupedData, we see that all functions like  avg max min and sum  return a DataFrame, so chaining is not possible.
Question: Here is my dataframe:    A B C     First row to delete row to shift   Second row to delete row to shift    And I want this output :    A B C     First row to shift    Second row to shift     I tried this code : df.shift(-1, axis=1)    A B C     row to delete row to shift    row to delete row to shift     The fact is, is there a way to keep the first column not modified ?

Answer: Be explicit, chose the columns to affect and reassign (or update): df[['B', 'C']] = df[['B', 'C']].shift(-1, axis=1, fill_value='')  Or: cols = ['B', 'C'] df[cols] = df[cols].shift(-1, axis=1, fill_value='') # or # df.update(df[cols].shift(-1, axis=1, fill_value=''))  Output:  A B C 0 First row to shift 1 Second row to shift  You can do the same per index/column: cols = ['B', 'C'] idx = [1] df.loc[idx, cols] = df.loc[idx, cols].shift(-1, axis=1, fill_value='') # or # df.update(df.loc[idx, cols].shift(-1, axis=1, fill_value=''))  Output:  A B C 0 First row row 1 Second row to shift