eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: the Polars user guide on Lists and Arrays explains how to manipulate Lists with common expression syntax using .list.eval(), i.e. how to operate on list elements. More specifically, the user guide states:  The function eval gives us access to the list elements and pl.element refers to each individual element, but we can also use pl.all() to refer to all of the elements of the list.  I do not understand the difference between using pl.element() vs pl.all(), i.e. when this distinction between individual and all elements mentioned in the quote becomes important. In the example below, both yield exactly the same result for various expressions. What am I missing? Thank you so much for your help! import polars as pl df = pl.DataFrame( { "a": [[1], [3,2], [6,4,5]] } ) print(df)  shape: (3, 1) ┌───────────┐ │ a │ │ --- │ │ list[i64] │ ╞═══════════╡ │ [1] │ │ [3, 2] │ │ [6, 4, 5] │ └───────────┘  ## using pl.element() result_element = df.with_columns( pl.col("a").list.eval(pl.element()**2).alias("square"), pl.col("a").list.eval(pl.element().rank()).alias("rank"), pl.col("a").list.eval(pl.element().count()).alias("count") ) ## using pl.all() result_all = df.with_columns( pl.col("a").list.eval(pl.all()**2).alias("square"), pl.col("a").list.eval(pl.all().rank()).alias("rank"), pl.col("a").list.eval(pl.all().count()).alias("count") ) print(result_element.equals(result_all))  True 

Answer: The method pl.all() called without arguments refers to all columns available in the context. It does not have a special meaning within list.eval(), but since the only column available inside of it are the list elements, it works the same as pl.element(). You could also get the same behavior using either  pl.col(''), a column whose name is an empty string. That is the name of the column (list) inside of .list.eval(...) and is equivalent to pl.element() pl.col('*'), a special name that selects all columns. This is equivalent to pl.all()  import polars as pl from polars.testing import assert_frame_equal df = pl.DataFrame({'x': [[1,2,3]]}) def test(expression): return pl.col('x').list.eval(expression.mul(2).add(1)) reference = df.select(test(pl.element())) for expression in [pl.all(), pl.col(''), pl.col('*')]: assert_frame_equal( reference, df.select(test(expression)) )
Question: I have this kind of dataframe: df = pd.DataFrame({ "A1": [1, 11, 111], "A2": [2, 22, 222], "A3": [3, 33, 333], "A4": [4, 44, 444], "A5": [5, 55, 555] }) A1 A2 A3 A4 A5 0 1 2 3 4 5 1 11 22 33 44 55 2 111 222 333 444 555  and this kind of mapping: mapping = { "A1": ["A2", "A3"], "A4": ["A5"] }  which means that I want all columns in list to have values from key column so: A2 and A3 should be populated with values from A1, and A5 should be populated with values from A4. Resulting dataframe should look like this:  A1 A2 A3 A4 A5 0 1 1 1 4 4 1 11 11 11 44 44 2 111 111 111 444 444  I managed to do it pretty simply like this: for k, v in mapping.items(): for col in v: df[col] = df[k]  but I was wondering if there is vectorized way of doing it (more pandactic way)?

Answer: You could rework the dictionary and use assign: out = df.assign(**{col: df.get(k) for k, v in mapping.items() for col in v})  NB. assign is not in place, either use this in chained commands, or reassign to df. Or you could reindex and rename/set_axis: dic = {v: k for k, l in mapping.items() for v in l} out = (df.reindex(columns=df.rename(columns=dic).columns) .set_axis(df.columns, axis=1) )  Output:  A1 A2 A3 A4 A5 0 1 1 1 4 4 1 11 11 11 44 44 2 111 111 111 444 444
Question: I am using the Zinc graph dataset via torch geometric which I access as zinc_dataset = ZINC(root='my_path', split='train')  Each data element is a graph zinc_dataset[0] looks like Data(x=[33, 1], edge_index=[2, 72], edge_attr=[72], y=[1])  I have computed a tensor valued feature for each graph in the dataset. I have stored these tensors in a list with the ith element of the list being the feature for the ith graph in zinc_dataset. I would like to add these new features to the data object. So ideally I want the result to be Data(x=[33, 1], edge_index=[2, 72], edge_attr=[72], y=[1], new_feature=[33,12])  I have looked at the solution provided by How to add a new attribute to a torch_geometric.data Data object element? but that hasn't worked for me. Could someone please help me take my list of new features and include them in the data object? Thanks

Answer: To add your list of new features (e.g. List[Tensor], with each tensor corresponding to a graph in the dataset) to each torch_geometric.data.Data object in a Dataset like ZINCYou can do this by simply assigning your new tensor as an attribute of each Data object. Here’s how you can do it step-by-step: import torch from torch_geometric.datasets import ZINC from torch_geometric.data import InMemoryDataset # 1. Load the ZINC training dataset zinc_dataset = ZINC(root='my_path', split='train') # 2. Create a list of new features for each graph # Replace this with your actual feature list (must match number of nodes per graph) new_features = [] for data in zinc_dataset: num_nodes = data.x.size(0) # data.x is [num_nodes, feature_dim] new_feat = torch.randn(num_nodes, 12) # Example: [num_nodes, 12] new_features.append(new_feat) # 3. Define a custom dataset that injects new_feature into each graph's Data object class ModifiedZINC(InMemoryDataset): def __init__(self, original_dataset, new_features_list): self.data_list = [] for i in range(len(original_dataset)): data = original_dataset[i] data.new_feature = new_features_list[i] self.data_list.append(data) super().__init__('.', transform=None, pre_transform=None) self.data, self.slices = self.collate(self.data_list) def __len__(self): return len(self.data_list) def get(self, idx): return self.data_list[idx] # 4. Create the modified dataset with new features modified_dataset = ModifiedZINC(zinc_dataset, new_features) # 5. Check the result sample = modified_dataset[0] print(sample) print("Shape of new feature:", sample.new_feature.shape)  output: Data(x=[33, 1], edge_index=[2, 72], edge_attr=[72], y=[1], new_feature=[33, 12]) Shape of new feature: torch.Size([33, 12])
Question: I want to compute the cumulative mean & std on a polars dataframe column. For the mean I tried this: import polars as pl df = pl.DataFrame({ 'value': [4, 6, 8, 11, 5, 6, 8, 15], 'class': ['A', 'A', 'B', 'A', 'B', 'A', 'B', 'B'] }) df.with_columns(cum_mean=pl.col('value').cum_sum().over('class') / pl.int_range(pl.len()).add(1).over('class'))  which correctly gives shape: (8, 3) ┌───────┬───────┬──────────┐ │ value ┆ class ┆ cum_mean │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ f64 │ ╞═══════╪═══════╪══════════╡ │ 4 ┆ A ┆ 4.0 │ │ 6 ┆ A ┆ 5.0 │ │ 8 ┆ B ┆ 8.0 │ │ 11 ┆ A ┆ 7.0 │ │ 5 ┆ B ┆ 6.5 │ │ 6 ┆ A ┆ 6.75 │ │ 8 ┆ B ┆ 7.0 │ │ 15 ┆ B ┆ 9.0 │ └───────┴───────┴──────────┘  However, this seems very clunky, and becomes a little more complicated (and possibly error-prone) for std. Is there a nicer (possibly built-in) version for computing the cum mean & cum std?

Answer: I might have a solution which is more clean. You can get to it using rolling-functions like rolling_mean or rolling_std. Here is my proposal: df.with_columns( cum_mean=pl.col('value').cum_sum().over('class')/pl.col('value').cum_count().over('class'), cum_mean_by_rolling=pl.col('value').rolling_mean(window_size=df.shape[0], min_samples=1).over('class'), cum_std_by_rolling=pl.col('value').rolling_std(window_size=df.shape[0], min_samples=1).over('class') )  If you define the window size as the number of rows in the data frame (df.shape[0]) and the minimum number of samples as 1, then you can get the wanted result. I also changed your implementation for the cum_mean a bit so that it is a bit shorter. If I run the code I get this result. shape: (8, 5) ┌───────┬───────┬──────────┬─────────────────────┬────────────────────┐ │ value ┆ class ┆ cum_mean ┆ cum_mean_by_rolling ┆ cum_std_by_rolling │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ f64 ┆ f64 ┆ f64 │ ╞═══════╪═══════╪══════════╪═════════════════════╪════════════════════╡ │ 4 ┆ A ┆ 4.0 ┆ 4.0 ┆ null │ │ 6 ┆ A ┆ 5.0 ┆ 5.0 ┆ 1.414214 │ │ 8 ┆ B ┆ 8.0 ┆ 8.0 ┆ null │ │ 11 ┆ A ┆ 7.0 ┆ 7.0 ┆ 3.605551 │ │ 5 ┆ B ┆ 6.5 ┆ 6.5 ┆ 2.12132 │ │ 6 ┆ A ┆ 6.75 ┆ 6.75 ┆ 2.986079 │ │ 8 ┆ B ┆ 7.0 ┆ 7.0 ┆ 1.732051 │ │ 15 ┆ B ┆ 9.0 ┆ 9.0 ┆ 4.242641 │ └───────┴───────┴──────────┴─────────────────────┴────────────────────┘  I did not find a more suitable build in function. Hope this helps.
Question: I want to combine two DataFrames of unequal length to a new DataFrame with the size of the larger one. Now, specifically, I want to pad the values of the shorter array by repeating it until it is large enough. I know this is possible for lists using itertools.cycle as follows: from itertools import cycle x = range(7) y = range(43) combined = zip(cycle(x), y)  Now I want to do the same for DataFrames: import pandas as pd df1 = pd.DataFrame(...) # length 7 df2 = pd.DataFrame(...) # length 43 df_comb = pd.concat([cycle(df1),df2], axis=1)  Of course this doesn't work, but I don't know if there is an option to do this or to just manually repeat the array.

Answer: If you want to combine the two DataFrames to obtain an output DataFrame of the length of the longest input with repetitions of the smallest input that restart like itertools.cycle, you could compute a common key (with numpy.arange and the modulo (%) operator) to perform a merge: out = (df1.merge(df2, left_on=np.arange(len(df1))%len(df2), right_on=np.arange(len(df2))%len(df1)) .drop(columns=['key_0']) )  Output:  col1 col2 col3 col4 0 A X a Y 1 B X b Y 2 C X c Y 3 D X a Y 4 E X b Y 5 F X c Y 6 G X a Y  Intermediate without dropping the merging key:  key_0 col1 col2 col3 col4 0 0 A X a Y 1 1 B X b Y 2 2 C X c Y 3 0 D X a Y 4 1 E X b Y 5 2 F X c Y 6 0 G X a Y  Used inputs: # df1 col1 col2 0 A X 1 B X 2 C X 3 D X 4 E X 5 F X 6 G X # df2 col3 col4 0 a Y 1 b Y 2 c Y
Question: I'm trying to be able to search chat groups by looking up the chatroom name. I'm using Django Q query... models.py class ChatGroup(models.Model): group_name = models.CharField(max_length=128, unique=True, default=shortuuid.uuid) groupchat_name = models.CharField(max_length=128, null=True, blank=True) picture = models.ImageField(upload_to='uploads/profile_pictures', default='uploads/profile_pictures/default.png', blank=True) about = models.TextField(max_length=500, blank=True, null=True) admin = models.ForeignKey(User, related_name='groupchats', blank=True, null=True, on_delete=models.SET_NULL) users_online = models.ManyToManyField(User, related_name='online_in_groups', blank=True) members = models.ManyToManyField(User, related_name='chat_groups', blank=True) is_private = models.BooleanField(default=False) def __str__(self): return self.group_name  views.py from django.db.models import Q class ChatSearch(View): def get(self, request, *args, **kwargs): query = self.request.GET.get('chat-query') chatroom_list = ChatGroup.objects.filter( Q(group_name__groupchat_name__icontains=query) ) context = { 'chatroom_list': chatroom_list } return render(request, 'chat/search.html', context)  I tried to add the Traceback but it was too much code for this post. Any help you can provide it would be greatly appreciate it!

Answer: According to the OP in a comment:  using a class based view was triggering a query when I opened the page. I had to create a new page with just the input query then use the query results on a separate page
Question: I have the code below which is built on top of ndb. When running I receive the two errors below. Can I ask for some guidance, specifically what is the connection_from_host referring to? import flask import config import util app = flask.Flask(__name__) from google.appengine.api import app_identity from google.appengine.api import taskqueue, search, memcache from apiclient.discovery import build, HttpError from google.cloud import ndb #from oauth2client.client import GoogleCredentials from apiclient.http import MediaIoBaseUpload from datetime import datetime, timedelta from functools import partial from io import BytesIO import os from os.path import splitext, basename from model import Config from model import VideosToCollections from pytz import timezone import datetime import httplib2 import iso8601 import time import requests import requests_toolbelt.adapters.appengine requests_toolbelt.adapters.appengine.monkeypatch() from operator import attrgetter import model from model import CallBack import re import config import google.appengine.api client = ndb.Client() def ndb_wsgi_middleware(wsgi_app): def middleware(environ, start_response): with client.context(): return wsgi_app(environ, start_response) return middleware app.wsgi_app = ndb_wsgi_middleware(google.appengine.api.wrap_wsgi_app(app.wsgi_app)) @app.route('/collectionsync/', methods=['GET']) #@ndb.transactional def collectionsync(): collection_dbs, collection_cursor = model.Collection.get_dbs( order='name' )  This returns:  /layers/google.python.pip/pip/lib/python3.12/site-packages/urllib3/contrib/appengine.py:111: AppEnginePlatformWarning: urllib3 is using URLFetch on Google App Engine sandbox instead of sockets. To use sockets directly instead of URLFetch see https://urllib3.readthedocs.io/en/1.26.x/reference/urllib3.contrib.html. google.api_core.exceptions.RetryError: Maximum number of 3 retries exceeded while calling <function make_call..rpc_call at 0x3ee3d42d6840>, last exception: 503 Getting metadata from plugin failed with error: '_AppEnginePoolManager' object has no attribute 'connection_from_host' 

Answer: I think the presence of requests_toolbelt dependency in your project caused the issue. It may have forced the requests library to use Google App Engine’s URLFetch service (urllib3), which requires URLFetch to be present. I think that was often necessary in the Python 2 runtime environment on GAE, but not in Python 3. You may try removing requests_toolbelt from your requirements.txt file (see this post). There’s also a possibility that the URLFetch warning is somewhat related to the RetryError connection_from_host. You should try migrating URLFetch to standard Python libraries compatible with the Python 3 GAE environment. Refer to this documentation for steps on replacing the URL Fetch API with a Python library, and you may also consider bypassing URLFetch. I hope this helps.
Question: I have HTML template in database column that is shared with another platform. The HTML template has placeholder variables. This value is pulled from DB in my Python script but for some reason, it is not replacing the placeholder variables with it's value. Here is what the HTML string that is in DB.  <html> Dear {FullName}, <p>We are excited to notify you that your account has been activated. Please login to your account at <a href="https://portal.example.com">My Account Portal</a>. </p> </html>  I get the name from DB table in the variable FullName. When the email is sent out, it doesn't replaces the name in the html template. If I create a local python variable with the same html template and not pull from database, it works just fine. So what I would like to know is how can I pull the html template from DB and make it work in my Python script? The DB template can't be updated to use %s in html template as it is a vendor provided system and changing that will break that application. Below is the Python script that I am using. cur.execute("select FirstName,LastName, Email, AlternateEmail,CustCode from Customer Where CustCode = ?", f"{CustCode}") for data in cur.fetchall(): FullName += data.FirstName+" "+data.LastName CustEmail += data.Email CustPIN += str(data.CustCode) ToEmail += data.AlternateEmail cur.execute("select Value from [dbo].[Configs] where [Name] = ?",'Email - Subject') for data in cur.fetchall(): mail_subject += data.Value cur.execute("select Value from [dbo].[Configs] where [Name] = ?",'Email - Message Body') for data in cur.fetchall(): mail_body = f''' {data.Value} ''' send_mail(from_mail='noreply@example.com', to_mail=f'{cust_email}',subject=f'{mail_subject}',mailbody=mail_body,mime_type='html')  if I change the script to hardcode the html template within the script, it works and I want to avoid that so I wouldn't have to change the script every time when the template changes. What are my option in this situation?

Answer: I think you just need to call the .format() string method on mail_body, and pass in the value of FullName. You can either do it at the end when you call send_mail(): mailbody=mail_body.format(FullName=FullName)  Or you can do it when you first read mail_body from the database: mail_body = f''' {data.Value} '''.format(FullName=FullName)
Question: There are two classes and I want to wrap item.foo method only once, to prevent cache = {param1: 'param_value'} being reinited class Foo: _count = 0 def foo(self, param2): self._count += param2 class Bar: _collection = [Foo(), Foo(), Foo()] def bar(self, param1, param2): for item in self._collection: wrapped_function = wrapper(item.foo, param1) wrapped_function(param2) def wrapper(func, param1): # some database call or whatever cache = {param1: 'param_value'} def _wrapper(*args, **kwargs): # access value to read print(cache[param1]) return func(*args, **kwargs) return _wrapper bar = Bar() bar.bar(1, 2)  It can be achieved if wrapper had no params in it and was used as a simple decorator, but I have to pass a param1 in it, though it's always the same. I also can save cache = {param1: 'param_value'} before the cycle in def bar and pass it as the parameter, but I was wondering if there any other way to accomplish it. Kinda can't wrap my head around it(pun intended)

Answer: It sounds like you want to wrap Foo.foo, not item.foo (which is different bound method for each value of foo. Something like class Bar: _collection = [Foo(), Foo(), Foo()] def bar(self, param1, param2): wrapped_function = wrapper(Foo.foo, param1) for item in self._collection: wrapped_function(item, param2)  It's more complicated if you only want to apply the wrapper once per unique type of object in _collection, as you need to build a cache of wrapped functions. Something like class Bar: _collection = [Foo(), Foo(), Foo()] wrap_cache = {} def bar(self, param1, param2): for item in self._collection: itype = type(item) if itype.foo not in wrap_cache: wrap_cache[itype] = wrapper(itype.foo, param1) wrapped_function = wrap_cache[itype] wrapped_function(param2)
Question: Scenario: I have a pandas series that contains 3 values. These values can vary between nan, 0 and any value above zero. I am trying to get the pct_change among the series whenever possible. Examples: [0,nan,50] [0,0,0] [0,0,50] [nan,nan,50] [nan,nan,0] [0,0,nan] [0,nan,0]  What I tried: from other SO questions I was able to come up with methods either trying to ignore the nan or shifting, but these can potentially yield a result with empty values. Ideally, if a result cannot be calculated, I would like to output a 0. Code tried: series_test = pd.Series([0,None,50]) series_test.pct_change().where(series_test.notna()) # tested but gives only NaN or inf series_test.pct_change(fill_method=None)[series_test.shift(2).notnull()].dropna() # tested but gives empty result  Question: What would be the correct way to approach this? Expected outputs: [0,nan,50] - 0 (undefined case) [0,0,0] - 0 (undefined case) [0,0,50] - 0 (undefined case) [nan,nan,50] - 0 (undefined case) [nan,nan,0] - 0 (undefined case) [0,0,nan] - 0 (undefined case) [0,nan,0] - 0 (undefined case) [1,nan,5] - 400% [0,1,5] - 400% [1,2,nan] - 100% [1,1.3,1.8] - 80% 

Answer: I think you could dropna, then compute the pct_change and only keep the max finite value: series_test.dropna().pct_change().loc[np.isfinite].max()  Or maybe: s.pct_change().where(np.isfinite, 0).max()  Example output for the second approach: [0, nan, 50] - 0.0 [0, 0, 0] - 0.0 [0, 0, 50] - 0.0 [nan, nan, 50] - 0.0 [nan, nan, 0] - 0.0 [0, 0, nan] - 0.0 [0, nan, 0] - 0.0 [1, nan, 5] - 4.0 [0, 1, 5] - 4.0 [0, 1, nan] - 0.0  Edit: given your comment, it looks like you want to use the first and last non-zero values to compute the percentage change. In this case, I'd use a custom function: def pct_chg(s): tmp = s[s>0] if len(tmp)>1: return (tmp.iloc[-1]-tmp.iloc[0])/tmp.iloc[0] return 0  Which should be equivalent to the more verbose: (series_test .where(s>0).bfill().ffill() .iloc[[0, -1]].pct_change().fillna(0).iloc[-1] )  Example: [0, nan, 50] - 0 [0, 0, 0] - 0 [0, 0, 50] - 0 [nan, nan, 50] - 0 [nan, nan, 0] - 0 [0, 0, nan] - 0 [0, nan, 0] - 0 [1, nan, 5] - 4.0 [0, 1, 5] - 4.0 [0, 1, nan] - 0 [1, 1.5, 1.6] - 0.6000000000000001
Question: I want to design a decorator that will allow the wrapped method to take a float or a numpy array of floats. If the passed argument was a float then a float should be returned and if it was a numpy array then a numpy array should be returned. Below is my MWE and latest attempt. I am using VSCode with pylance version v2024.3.2 and Python version 3.12.3. I have a large number of functions I'd like to apply this decorator to. If I was only dealing with one or two functions I could do away with the decorator approach entirely and use @overload. The type error I get is the following: Argument of type "float" cannot be assigned to parameter "a" of type "NDArray[float64]" in function "func" "float" is incompatible with "NDArray[float64]"  import numpy as np from numpy import float64 from numpy.typing import NDArray from collections.abc import Callable from typing import TypeAlias, TypeVar T = TypeVar('T', float, NDArray[float64]) PreWrapFunc: TypeAlias = Callable[[NDArray[float64]], NDArray[float64]] PostWrapFunc: TypeAlias = Callable[[T], T] def my_decorator(method: PreWrapFunc) -> PostWrapFunc: def wrapper(arg: T) -> T: if isinstance(arg, float): result = method(np.array([arg,])) return result[0] else: return method(arg) return wrapper @my_decorator def func(a: NDArray[float64]) -> NDArray[float64]: return a * 2 func(1.0) # 2.0, type error is happening here! func(np.array([1.0,])) # array([2.]) 

Answer: Define a protocol with overloaded signatures for your wrapper and then use that as the return type of your decorator: import numpy as np from numpy import float64 from numpy.typing import NDArray from typing import Callable, Protocol, overload class AsFloatOrArray(Protocol): @overload def __call__(self, arg: float) -> float: ... @overload def __call__(self, arg: NDArray[float64]) -> NDArray[float64]: ... def __call__(self, arg): ... def my_decorator( method: Callable[[NDArray[float64]], NDArray[float64]], ) -> AsFloatOrArray: @overload def wrapper(arg: float) -> float: ... @overload def wrapper(arg: NDArray[float64]) -> NDArray[float64]: ... def wrapper(arg): if isinstance(arg, float): return method(np.array([arg]))[0] else: return method(arg) return wrapper @my_decorator def func(a: NDArray[float64]) -> NDArray[float64]: return a * 2 print(func(1.0)) print(func(np.array([1.0])))  Or, more concisely, write the decorator as a class and provide overloaded signatures for the __call__ method: import numpy as np from numpy import float64 from numpy.typing import NDArray from typing import Callable, overload class MyDecorator: def __init__( self, method: Callable[[NDArray[float64]], NDArray[float64]], ) -> None: self.method = method @overload def __call__(self, arg: float) -> float: ... @overload def __call__(self, arg: NDArray[float64]) -> NDArray[float64]: ... def __call__(self, arg): if isinstance(arg, float): return self.method(np.array([arg]))[0] else: return self.method(arg) @MyDecorator def func(a: NDArray[float64]) -> NDArray[float64]: return a * 2 print(func(1.0)) print(func(np.array([1.0])))