eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to check if string_1 = "this example string" contains a column value as a substring. For example the first value in Col B should be True since "example" is a substring of string_1 string_1 = "this example string" df = pl.from_repr(""" ┌────────┬─────────┬────────────┐ │ Col A ┆ Col B ┆ Col C │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞════════╪═════════╪════════════╡ │ 448220 ┆ example ┆ 7101936801 │ │ 518398 ┆ 99999 ┆ 9999900091 │ │ 557232 ┆ 424570 ┆ 4245742060 │ └────────┴─────────┴────────────┘ """)  This is what I have tried so far, but it's returning the following error: df=df.with_columns(pl.col("Col B").apply(lambda x: x in string_1).alias("new_col"))  AttributeError: 'Expr' object has no attribute 'apply' 

Answer: It's always better to avoid using python functions and use native polars expressions.  pl.lit() to create a dummy column from string_1 str.contains() to check if string contains a column value.  ( df .with_columns(pl.lit(string_1).str.contains(pl.col('Col B')).alias('new_col') )  Or you can use name.keep() if you want to check all columns. ( df .with_columns(pl.lit(string_1).str.contains(pl.all()).name.keep()) ) ┌───────┬───────┬───────┐ │ Col A ┆ Col B ┆ Col C │ │ --- ┆ --- ┆ --- │ │ bool ┆ bool ┆ bool │ ╞═══════╪═══════╪═══════╡ │ false ┆ true ┆ false │ │ false ┆ false ┆ false │ │ false ┆ false ┆ false │ └───────┴───────┴───────┘  or something like this if you need all new columns:  name.suffix()  ( df .with_columns(pl.lit(string_1).str.contains(pl.all()).name.suffix('_match')) ) ┌────────┬─────────┬────────────┬─────────────┬─────────────┬─────────────┐ │ Col A ┆ Col B ┆ Col C ┆ Col A_match ┆ Col B_match ┆ Col C_match │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ bool ┆ bool ┆ bool │ ╞════════╪═════════╪════════════╪═════════════╪═════════════╪═════════════╡ │ 448220 ┆ example ┆ 7101936801 ┆ false ┆ true ┆ false │ │ 518398 ┆ 99999 ┆ 9999900091 ┆ false ┆ false ┆ false │ │ 557232 ┆ 424570 ┆ 4245742060 ┆ false ┆ false ┆ false │ └────────┴─────────┴────────────┴─────────────┴─────────────┴─────────────┘
Question: I am trying to create a Polars DataFrame that includes a column of structs based on another DataFrame column. Here's the setup: import polars as pl df = pl.DataFrame( [ pl.Series("start", ["2023-01-01"], dtype=pl.Date).str.to_date(), pl.Series("end", ["2024-01-01"], dtype=pl.Date).str.to_date(), ] )  shape: (1, 2) ┌────────────┬────────────┐ │ start ┆ end │ │ --- ┆ --- │ │ date ┆ date │ ╞════════════╪════════════╡ │ 2023-01-01 ┆ 2024-01-01 │ └────────────┴────────────┘  df = df.with_columns( pl.date_range(pl.col("start"), pl.col("end"), "1mo", closed="left") .implode() .alias("date_range") )  shape: (1, 3) ┌────────────┬────────────┬─────────────────────────────────┐ │ start ┆ end ┆ date_range │ │ --- ┆ --- ┆ --- │ │ date ┆ date ┆ list[date] │ ╞════════════╪════════════╪═════════════════════════════════╡ │ 2023-01-01 ┆ 2024-01-01 ┆ [2023-01-01, 2023-02-01, … 202… │ └────────────┴────────────┴─────────────────────────────────┘  Now, I want to make a struct out of the year/month parts: df = df.with_columns( pl.col("date_range") .list.eval( pl.struct( { "year": pl.element().dt.year(), "month": pl.element().dt.month(), } ) ) .alias("years_months") )  But this does not work. Maybe I ought not to implode the date_range's output into a list, but I am not sure how to create a struct directly from its result either. My best idea is one I don't like because I have to repeatedly call pl.list.eval: df = ( df.with_columns( pl.col("date_range").list.eval(pl.element().dt.year()).alias("year"), pl.col("date_range").list.eval(pl.element().dt.month()).alias("month"), ) .drop("start", "end", "date_range") .explode("year", "month") .select(pl.struct("year", "month")) ) df  The other idea is to use map_elements, but I think that ought to be something of a last resort. What's the idiomatic way to eval into a struct?

Answer: You shouldn't be passing a dictionary into the struct constructor. Just pass each IntoExpr as a keyword argument - that is, pl.struct(key1=IntoExprA, key2=IntoExprB, ...). In your case, you could replace your struct-ification code with the following: df = df.with_columns( pl.col("date_range") .list.eval( pl.struct( year = pl.element().dt.year(), month = pl.element().dt.month() ) ) .alias("years_months") )  Printing df gave me this output: ┌────────────┬────────────┬─────────────────────────────────┬─────────────────────────────────┐ │ start ┆ end ┆ date_range ┆ years_months │ │ --- ┆ --- ┆ --- ┆ --- │ │ date ┆ date ┆ list[date] ┆ list[struct[2]] │ ╞════════════╪════════════╪═════════════════════════════════╪═════════════════════════════════╡ │ 2023-01-01 ┆ 2024-01-01 ┆ [2023-01-01, 2023-02-01, … 202… ┆ [{2023,1}, {2023,2}, … {2023,1… │ └────────────┴────────────┴─────────────────────────────────┴─────────────────────────────────┘  If your dictionary was constructed earlier in the program, you can use unpacking to get the kwargs back out: df = df.with_columns( pl.col("date_range") .list.eval( pl.struct(**example_dict) ) .alias("years_months") )
Question: I m interested in seeing neural network as graph using tensorboard. I have constructed a network in pytorch with following code- import torch BATCH_SIZE = 16 DIM_IN = 1000 HIDDEN_SIZE = 100 DIM_OUT = 10 class TinyModel(torch.nn.Module): def __init__(self): super(TinyModel, self).__init__() self.layer1 = torch.nn.Linear(DIM_IN, HIDDEN_SIZE) self.relu = torch.nn.ReLU() self.layer2 = torch.nn.Linear(HIDDEN_SIZE, DIM_OUT) def forward(self, x): x = self.layer1(x) x = self.relu(x) x = self.layer2(x) return x some_input = torch.randn(BATCH_SIZE, DIM_IN, requires_grad=False) ideal_output = torch.randn(BATCH_SIZE, DIM_OUT, requires_grad=False) model = TinyModel()  Setting-up tensorboard from torch.utils.tensorboard import SummaryWriter # Create a SummaryWriter writer = SummaryWriter("checkpoint") # Add the graph to TensorBoard writer.add_graph(model, some_input) writer.close()  While I run tensorboard --logdir=checkpoint on terminal , I receive the following error - Traceback (most recent call last): File "/home/k/python_venv/bin/tensorboard", line 5, in <module> from tensorboard.main import run_main File "/home/k/python_venv/lib/python3.10/site-packages/tensorboard/main.py", line 27, in <module> from tensorboard import default File "/home/k/python_venv/lib/python3.10/site-packages/tensorboard/default.py", line 39, in <module> from tensorboard.plugins.hparams import hparams_plugin File "/home/k/python_venv/lib/python3.10/site-packages/tensorboard/plugins/hparams/hparams_plugin.py", line 30, in <module> from tensorboard.plugins.hparams import backend_context File "/home/k/python_venv/lib/python3.10/site-packages/tensorboard/plugins/hparams/backend_context.py", line 26, in <module> from tensorboard.plugins.hparams import metadata File "/home/k/python_venv/lib/python3.10/site-packages/tensorboard/plugins/hparams/metadata.py", line 32, in <module> NULL_TENSOR = tensor_util.make_tensor_proto( File "/home/k/python_venv/lib/python3.10/site-packages/tensorboard/util/tensor_util.py", line 405, in make_tensor_proto numpy_dtype = dtypes.as_dtype(nparray.dtype) File "/home/k/python_venv/lib/python3.10/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py", line 677, in as_dtype if type_value.type == np.string_ or type_value.type == np.unicode_: File "/home/k/python_venv/lib/python3.10/site-packages/numpy/__init__.py", line 397, in __getattr__ raise AttributeError( AttributeError: `np.string_` was removed in the NumPy 2.0 release. Use `np.bytes_` instead.. Did you mean: 'strings'?  Probably the issue will be fixed in future releases, but is there a fix for now?

Answer: Your code seems to be written in numpy < 2.0 compatible way, but it looks like you are running it with numpy >=2. Try downgrading the numpy version to < 2.0.
Question: z = torch.tensor(1, dtype= torch.int64) z[:] = 5 Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: slice() cannot be applied to a 0-dim tensor.  I'm trying to assign a value to a torch tensor but because it has zero dimensions the slice operator doesn't work. How do I assign a new value then?

Answer: This is actually possible with copy_: z = torch.tensor(1, dtype=torch.int32) z.copy_(3.) >>> tensor(3, dtype=torch.int32)  This will also cast it to the dtype of my variable.
Question: When I run a function in a python package, it returns: import gala.potential as gp Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/homebrew/lib/python3.11/site-packages/gala/potential/__init__.py", line 1, in <module> from .potential import * File "/opt/homebrew/lib/python3.11/site-packages/gala/potential/potential/__init__.py", line 2, in <module> from .cpotential import * ImportError: dlopen(/opt/homebrew/lib/python3.11/site-packages/gala/potential/potential/cpotential.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_gsl_sf_gamma'  As the error message contains "_gsl_sf_gamma", so I suppose the problem is coming from the GSL library of C/C++. I am using Mac with M2 Chip. I have installed GSL before by homebrew, so the path of GSL is /opt/homebrew/Cellar/gsl/2.7.1  Moreover, I have also installed gcc before by homebrew. However, the default gcc is installed from Xcode, which uses clang and its path is /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin  I suppose the problem is from the inconsistency between the GSL and the gcc. I have tried to reinstall GSL through conda (without uninstall the previous GSL), but it doesn't work.

Answer: OK so this is a linking problem. The library you are trying to import tries to call a function _gsl_sf_gamma (probably from the GSL library) but cannot find it. The library you are trying to import is programmed in a compiled language (most likely C or C++). In these languages, libraries, work a bit differently from how they work in python. There are two ways to get functions or other values reffered to by names (symbols) from a library.  Static linking bundles the function from the library in the binary calling it. Dynamic linking loads the library on-demand, like python does with its import statement. It is more complicated because the binary calling the function needs to find the library somewhere on the system.  So basically something was not properly linked. Probably that that cpotential.cpython-311-darwin.so expects the GSL library to already be dynamically loaded. Now the conda environment changes a bunch of things, including where programs are going to look for dynamically linked libraries. It probably loaded a different version of cpotential.cpython-311-darwin.so which loaded GSL correctly from conda's libraries
Question: I am using the following custom feature extractor for my StableBaselines3 model: import torch.nn as nn from stable_baselines3 import PPO class Encoder(nn.Module): def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim=2): super(Encoder, self).__init__() self.encoder = nn.Sequential( nn.Linear(input_dim, embedding_dim), nn.ReLU() ) self.regressor = nn.Sequential( nn.Linear(embedding_dim, hidden_dim), nn.ReLU(), ) def forward(self, x): x = self.encoder(x) x = self.regressor(x) return x model = Encoder(input_dim, embedding_dim, hidden_dim) model.load_state_dict(torch.load('trained_model.pth')) # Freeze all layers for param in model.parameters(): param.requires_grad = False class CustomFeatureExtractor(BaseFeaturesExtractor): def __init__(self, observation_space, features_dim): super(CustomFeatureExtractor, self).__init__(observation_space, features_dim) self.model = model # Use the pre-trained model as the feature extractor self._features_dim = features_dim def forward(self, observations): features = self.model(observations) return features policy_kwargs = { "features_extractor_class": CustomFeatureExtractor, "features_extractor_kwargs": {"features_dim": 64} } model = PPO("MlpPolicy", env=envs, policy_kwargs=policy_kwargs)  The model is trained well so far with no issues and good results. Now I want to not freeze the weights, and try to train the Feature Extractor as well starting from the initial pre-trained weight. How can I do that with such a custom Feature Extractor defined as a class inside another class? My feature extractor is not the same as the one defined in the documentation, so I am not sure if it will be trained. Or will it start training if I unfreeze the layers?

Answer: UPDATED answer Because your CustomFE imports already freezer Encoder (with requires_grad = False) you have that kind of situation where all weights of CustomFE are frozen. Thus by default CustomFE is not trainable. You will need to unfreeze it manually:  model = PPO("MlpPolicy", env='FrozenLake8x8', policy_kwargs=policy_kwargs) # get model feature extractor feature_extr: CustomFeatureExtractor = model.policy.features_extractor # convert all parameters to trainable for name, param in feature_extr.named_parameters(): param.requires_grad = True # check parameters before training encoder = feature_extr.model.encoder for name, param in encoder[0].named_parameters(): print(name, param.mean()) # train the model model.learn(total_timesteps = 5) # check parameters after training (if mean changed parameters are training) feature_extr: CustomFeatureExtractor = model.policy.features_extractor encoder = feature_extr.model.encoder for name, param in encoder[0].named_parameters(): print(name, param.mean())
Question: Let  be independent normal random variables with means  and unit variances, i.e.   I would like to compute the probability  Is there any easy way to compute the above probability using scipy.stats.multivariate_normal? If not, how do we do it using scipy.integrate?

Answer: Based on the info from one of SO post mentioned in the comments: https://math.stackexchange.com/questions/270745/compute-probability-of-a-particular-ordering-of-normal-random-variables Also same method mentioned on wiki here: https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Affine_transformation You should convert F(X) to F(Y) in the following way: import numpy as np from scipy.stats import norm, multivariate_normal mvn = multivariate_normal def compute_probability(thetas): """ following: https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Affine_transformation F(X1<X2<X3<...<Xn) = F(Y1<0, Y2<0, ... Y_{n-1}<0) where: Y_i = X_{i-1} - X_{i} ... or what is the same: Y = c + BX mean and sigma of Y can be found based on formulas from wikipedia """ n = len(thetas) # set diagonal to 1 B = np.eye(n) # set right to diagonal to -1 idx = np.arange(n-1) B[idx,idx+1] = -1 # remove last row B = B[:-1] # calculate multivate sigma and mu sigma = np.eye(n) mu_new = B.dot(thetas.T) sigma_new = B.dot(sigma).dot(B.T) MVN = mvn(mean=mu_new, cov = sigma_new) x0 = np.zeros(shape = n-1) p = MVN.cdf(x = x0) return p # Example usage: theta = np.array([1, -2, 0.5]) # Example coefficient p = compute_probability(theta) print(p)  Outputs: theta = (0,0) p = 0.5 theta = (0,0,0) p = 0.1666 theta = (100, -100) p = 0 theta = (-100, 100) p = 1 theta = (0,0,0,100, -100) p = 0
Question: My code: df = pd.read_xml( path_or_buffer=PATH, xpath="//Data", compression="gzip" )  I'm using Pandas read_xml() function to read xml.gz format data. I'm using Pandas 1.3.2 version. When I tried to read the data, Pandas read data wrongly. The data looks like as below. Both colA and colB should be a string. 1st data file: <Data> <colA>abc</colA> <colB>168E3</colB> </Data> <Data> <colA>def</colA> </Data>  2nd data file: <Data> <colA>ghi</colA> <colB>23456</colB> </Data> <Data> <colA>jkl</colA> </Data>  When I use read_xml() function, it looks like below: 1st dataframe: colA: abc, def colB: 168000.0, None  2nd dataframe: colA: ghi, jkl colB: 23456.0, None  I want to read the data in string format but there is no dtype argument in pandas 1.3.2. I want to know:  How can I read the data with designated data type? When there is missing data in a column, Pandas will assign the float type to that column. How to avoid it, or is there any setting to configure the data type of column with missing value when data is read?  Please note that I can only this Pandas version and can't update it.

Answer: Applying a simple xslt transformation to force string type on every value could work (I don't have pandas 1.3.2 to test but works on pandas 2.x). It does not seem possible to obtain a canonical answer given the version contraint. An alternative could be to parse xml doc with lxml and manually populate the dataframe. The transformation concatenates a non number character that is later removed XSLT <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="node()"> <xsl:copy> <xsl:apply-templates select="node()"/> <xsl:value-of select="'|'"/> </xsl:copy> </xsl:template> </xsl:stylesheet>  Python import pandas as pd from io import StringIO xml_ ="""\ <root> <Data> <colA>abc</colA> <colB>168E3</colB> </Data> <Data> <colA>def</colA> </Data> </root>""" file = StringIO(xml_) df = pd.read_xml(xml_, stylesheet="/home/lmc/tmp/test.xslt" ) print(df.apply(lambda x: x.str[:-1]))  Result  colA colB 0 abc 168E3 1 def None  Original result  colA colB 0 abc| 168E3| 1 def| None
Question: I have a config file for a python script that stores multiple different values my config.ini looks like this: [DHR] key1 = "\\path1\..." key2 = "\\path2\..." key3 = "\\path3\file-{today}.xlsx"  my .py has a date variable that gets today's date like: today = str(date.today().strftime('%y%m%d'))  However when I read the .ini file the variable does not get appended to the value as I expected. print(config.read("config.ini")) "\\path3\file-{today}.xlsx"  How can I adjust my script to append the variable to the path so that it looks like this: "\\path3\file-240709.xlsx" 

Answer: You could provide a filtered locals() dictionary to the parser to interpolate variables. However, the syntax of the ini needs to be changed to configparser's basic interpolation syntax: [DHR] key3 = "\\path3\file-%(today)s.xlsx"  from configparser import ConfigParser from datetime import date today = str(date.today().strftime('%y%m%d')) # instantiate parser with local variables, filtered for string-only values parser = ConfigParser({k: v for k, v in locals().items() if isinstance(v, str)}) parser.read('config.ini') print(parser["DHR"]["key3"]) # => "\\path3\file-240710.xlsx"
Question: I want to know exactly what I can do in spark without triggering the computation of the spark RDD/DataFrame. It's my understanding that only actions trigger the execution of the transformations in order to produce a DataFrame. The problem is that I'm unable to find a comprehensive list of spark actions. Spark documentation lists some actions, but it's not exhaustive. For example show is not there, but it is considered an action.  Where can I find a full list of actions? Can I assume that all methods listed here are also actions? 

Answer: All the methods annotated in the with @group action are actions. They can be found as a list here in scaladocs. They can also be found in the source where each method is defined, looking like this:  * @group action * @since 1.6.0 */ def show(numRows: Int): Unit = show(numRows, truncate = true)  Additionally, some other methods do not have that annotation, but also perform an eager evaluation: Those that call withAction. Checkpoint, for example, actually performs an action but isn't grouped as such in the docs: private[sql] def checkpoint(eager: Boolean, reliableCheckpoint: Boolean): Dataset[T] = { val actionName = if (reliableCheckpoint) "checkpoint" else "localCheckpoint" withAction(actionName, queryExecution) { physicalPlan => val internalRdd = physicalPlan.execute().map(_.copy()) if (reliableCheckpoint) {  To find all of them  Go to the source Use control + F Search for private def withAction Click on withAction On the right you should see a list of methods that use them. This is how that list currently looks:
Question: I have the following endpoint: @router.get('/events/{base_id}') def asd(base_id: common_input_types.event_id) -> None: do_something()  And this is common_input_types.event_id: event_id = Annotated[ int, fastapi.Depends(dependency_function), fastapi.Path( example=123, description="Lore", gt=5, le=100, ), ]  This does not trigger the dependency function. When I remove the fastapi.Path part, it works. But then I won't have the example and the min-max in swagger. How do I specify both?

Answer: I don't think it's possible to annotate a parameter with Path and Depends at the same time. Depending on the logic you want to implement, you can move this logic to dependency_function: from typing import Annotated import fastapi app = fastapi.FastAPI() router = fastapi.APIRouter(prefix="") async def dependency_function( base_id: Annotated[ int, fastapi.Path(example=123, description="Lore", gt=5, le=100) ] ): some_condition = True if some_condition: return 123 else: return base_id @router.get('/events/{base_id}') def asd(base_id: Annotated[int, fastapi.Depends(dependency_function)]) -> None: print(base_id) app.include_router(router)
Question: I am attempting to perform on an integral from -infinity to infinity, with an equation containing a term that is raised to the power M. At very high (5000+) values of M, this value of this term can exceed 10^100. I have this python (3.10) program, which works for M = 1000, but nothing much beyond that. import numpy as np from scipy import integrate, special def integrand(u, M, Z, r): sqrt_r = np.sqrt(r) sqrt_term = np.sqrt(2 * (1 - r)) arg1 = (Z + sqrt_r * u) / sqrt_term arg2 = (-Z + sqrt_r * u) / sqrt_term erf_term = special.erf(arg1) - special.erf(arg2) return np.exp(-u**2 / 2) / np.sqrt(2 * np.pi) * erf_term**M def evaluate_integral(M, Z, r): result, _ = integrate.quad(integrand, -np.inf, np.inf, args=(M, Z, r)) return 1 / (2**M) * result # Example usage: M = 1000 Z = 5 r = 0.6 integral_value = evaluate_integral(M, Z, r)  The error I get is: IntegrationWarning: The occurrence of roundoff error is detected, which prevents the requested tolerance from being achieved. The error may be underestimated. result, _ = integrate.quad(integrand, -np.inf, np.inf, args=(M, Z, r))  and the result for integral_value is nan. Is there a way I can perform this integral using Python, which would be convenient as it allows me to be included with the rest of my program, or will I be forced to use a different tool?

Answer: No, your only problem is that you took the factor 1/2**M outside of the integral. If you put it back inside you will then have (erf_term/2)**M  and that will not blow up because the difference of two error functions cannot exceed 2, so the new bracketed factor is less than 1.
Question: I have this sparse vector val with this Pythonic way: idx = [2, 5, 6] val = [69, 12, 15] _ = np.zeros(idx[-1]+1) for i in idx: _[i] = val[idx.index(i)] print(_) dbg(_)  Here is the output: [ 0. 0. 69. 0. 0. 12. .15] (6,) float64  How do I achieve my goal where it's not using for loops, instead using a Numpythonic way?

Answer: idx = [2, 5, 6] val = [69, 12, 15] _ = np.zeros(idx[-1]+1)  numpy lets you set multiple values at once _[idx] = val
Question: So I need to make take some derivatives first and later in the code (when the explicit form of f(x) is given) substitute the Derivative(f(x), x) with the explicit form of the derivative import sympy as sp import math x = sp.symbols('x') f = sp.Function('f')('x') df = f.diff(x) f = sp.sin(x) print(df)  What I expect to see as an output is cos(x) and not Derivative(f(x), x) that it gives me anyway. I tried subs and doit with no success. What would be a proper way of getting what I need?

Answer: Substitute your known function into the derivative and execute the doit method: import sympy as sp import math x = sp.symbols('x') f = sp.Function('f')('x') df = f.diff(x) f_known = sp.sin(x) df.subs(f, f_known).doit()