eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I use the google API for a personal project, so I don't have my app verified with google. I use exactly this code example myself and a token.json file gets generated when logging in. Everything works fine, the access token changes more or less every time I make a request (every 10 min). After a week, the request fails. Exactly a week after the "expiry" field in the token.json file. google.auth.exceptions.RefreshError: ('invalid_grant: Token has been expired or revoked.', {'error': 'invalid_grant', 'error_description': 'Token has been expired or revoked.'})  If I understand everything correctly, google should update refresh_token as well, but this did not happen. I thought this part would handle getting a new refresh token: if os.path.exists('token.json'): creds = Credentials.from_authorized_user_file('token.json', SCOPES) # If there are no (valid) credentials available, let the user log in. if not creds or not creds.valid: if creds and creds.expired and creds.refresh_token: creds.refresh(Request()) ## HERE ## else:  Might this be caused by my app not being verified? I did not find any information and hardly none about the behavior of the refresh_token.

Answer: You need to publish your app to production in order to remove the 7 days limitation. In APIs & Services / Oauth consent screen:  From google documentation about refresh token expiration:  A Google Cloud Platform project with an OAuth consent screen configured for an external user type and a publishing status of "Testing" is issued a refresh token expiring in 7 days.  Also about Testing publishing status:  Projects configured with a publishing status of Testing are limited to up to 100 test users listed in the OAuth consent screen. A test user consumes a project's test user quota once added to the project. Google will display a warning message before allowing a specified test user to authorize scopes requested by your project's OAuth clients. The warning message confirms the user has test access to your project but should consider the risks associated with granting access to their data to an unverified app. Authorizations by a test user will expire seven days from the time of consent. If your OAuth client requests an offline access type and receives a refresh token, that token will also expire. A Brand Account may authorize scopes requested by your project's OAuth clients if a specified test user manages the Brand Account. A test user may be unable to authorize scopes requested by your project's OAuth clients due to the availability of Google Services for the account or configured restrictions. A Google Workspace may control which third-party apps access its data or an account enrolled in Advanced Protection may block most non-Google apps.
Question: I am trying to return my model with the defined field names instead of its aliases. class FooModel(BaseModel): foo: str = Field(..., alias="bar") @app.get("/") -> FooModel: return FooModel(**{"bar": "baz"})  The response will be {"bar": "baz"} while I want {"foo": "baz"}. I know it's somewhat possible when using the dict method of the model, but it doesn't feel right and messes up the typing of the request handler. @app.get("/") -> FooModel: return FooModel(**{"bar": "baz"}).dict(by_alias=False)  I feel like it should be possible to set this in the config class, but I can't find the right option.

Answer: You can add response_model_by_alias=False to path operation decorator. This key is mentioned here in the documentation. For example: @app.get("/model", response_model=Model, response_model_by_alias=False) def read_model(): return Model(alias="Foo")
Question: Example: from typing import TypedDict class MyType(TypedDict): a: int b: int t = MyType(a=1, b=2) t.update(b=3)  mypy toy.py complains toy.py:9:1: error: Unexpected keyword argument "b" for "update" of "TypedDict" Found 1 error in 1 file (checked 1 source file) 

Answer: It appears this is a known open issue for mypy: https://github.com/python/mypy/issues/6019 For now if you want mypy to not bother you with this error, you'll need to tell it to ignore it: t.update(b=3) # type: ignore[call-arg]
Question: One of my friends asked me about this piece of code: array = [1, 8, 15] gen = (x for x in array if array.count(x) > 0) array = [2, 8, 22] print(list(gen))  The output: [8]  Where did the other elements go?

Answer: The answer is in the PEP of the generator expressions, in particular the session Early Binding vs Late biding:  After much discussion, it was decided that the first (outermost) for-expression should be evaluated immediately and that the remaining expressions be evaluated when the generator is executed.  So basically the array in: x for x in array  is evaluated using the original list [1, 8, 15] (i.e. immediately), while the other one: if array.count(x) > 0  is evaluated when the generator is executed using: print(list(gen))  at which point array refers to a new list [2, 8, 22]
Question: Suppose I have a Rust struct like this struct X{...} struct Y{ x:X }  I'd like to be able to write python code that accesses X through Y y = Y() y.x.some_method()  What would be the best way to implement it in PyO3? Currently I made two wrapper classes #[pyclass] struct XWrapper{ x:X } #[pyclass] struct YWrapper{ y:Y } #[pymethods] impl YWrapper{ #[getter] pub fn x(&self)->XWrapper{ XWrapper{x:self.y.clone()} } }  However, this requires clone(). I'd rather want to return reference. Of course I know that if X was a pyclass, then I could easily return PyRef to it. But the problem is that X and Y come from a Rust library and I cannot nilly-wily add #[pyclass] to them.

Answer: I don't think what you say is possible without some rejigging of the interface: Your XWrapper owns the x and your Y owns its x as well. That means creating an XWrapper will always involve a clone (or a new). Could we change XWrapper so that it merely contains a reference to an x? Not really, because that would require giving XWrapper a lifetime annotation, and PyO3 afaik doesn't allow pyclasses with lifetime annotation. Makes sense, because passing an object to python puts it on the python heap, at which point rust loses control over the object. So what can we do? Some thoughts: Do you really need to expose the composition structure of y to the python module? Just because that's the way it's organized within Rust doesn't mean it needs to be that way in Python. Your YWrapper could provide methods to the python interface that behind the scenes forward the request to the x instance: #[pymethods] impl YWrapper{ pub fn some_method(&self) { self.y.x.some_method(); } }  This would also be a welcome sight to strict adherents of the Law of Demeter ;) I'm trying to think of other clever ways. Depending on some of the details of how y.x is accessed and modified by the methods of y itself, it might be possible to add a field x: XWrapper to the YWrapper. Then you create the XWrapper (including a clone of y.x) once when YWrapper is created, and from then on you can return references to that XWrapper in your pub fn x. Of course that becomes much more cumbersome when x gets frequently changed and updated via the methods of y... In a way, this demonstrates the clash between Python's ref-counted object model and Rust's ownership object model. Rust enforces that you can't arbitrarily mess with objects unless you're their owner.
Question: I tried to learn featuretools following documentation from featuretools.com. A error came up: AttributeError: 'EntitySet' object has no attribute 'entity_from_dataframe' Could you help me? Thank you. Code:  import featuretools as ft data = ft.demo.load_mock_customer() transactions_df = data["transactions"].merge(data["sessions"]).merge(data["customers"]) transactions_df.sample(10) products_df = data["products"] products_df es = ft.EntitySet(id="customer_data") es = es.entity_from_dataframe(entity_id="transactions", dataframe=transactions_df, index="transaction_id", time_index="transaction_time", variable_types={"product_id": ft.variable_types.Categorical, "zip_code": ft.variable_types.ZIPCode}) es  Code source: https://docs.featuretools.com/en/v0.16.0/loading_data/using_entitysets.html#creating-entity-from-existing-table

Answer: The documentation you are using is for an older version of Featuretools. You can find the updated Getting Started documentation that works with Featuretools version 1.0 here: https://featuretools.alteryx.com/en/stable/getting_started/getting_started_index.html
Question: I execute the below command, in the same working directory as file worker.py: poetry run worker.py  Terminal: me@LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/composite_key/compositekey$ poetry run worker.py FileNotFoundError [Errno 2] No such file or directory: b'/snap/bin/worker.py' at /usr/lib/python3.8/os.py:601 in _execvpe 597│ path_list = map(fsencode, path_list) 598│ for dir in path_list: 599│ fullname = path.join(dir, file) 600│ try: → 601│ exec_func(fullname, *argrest) 602│ except (FileNotFoundError, NotADirectoryError) as e: 603│ last_exc = e 604│ except OSError as e: 605│ last_exc = e me@LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/composite_key/compositekey$ ls Citizenship.csv __pycache__ dagster.yaml pytest.ini simulate_alien_dict.py tasks.py "Gordian Algorithms' Times.xlsx" config.yaml data run_pipeline.yaml simulate_data.ipynb tests __init__.py currency_symbols_map.json modules simulate_alien_dict.ipynb simulate_data.py worker.py  Clearly, we see the file is there (bottom-right).  Questions  Why is this an issue caused? How would you resolve this issue, in future?  Please let me know if there's anything else I should add to post

Answer: poetry run means "run the following command in the venv managed by poetry". So the correct way of using it in your case is: poetry run python worker.py
Question: Below is a sample JSON: { "School": [ {"@id": "ABC_1", "SchoolType": {"@tc": "10023204", "#text": "BLUE FOX"}}, {"@id": "ABC_2", "SchoolType": {"@tc": "143", "#text": "AN EAGLE"}}, {"@id": "ABC_3", "SchoolType": {"@tc": "21474836", "#text": "OTHER REASONS"}, "SchoolStatus": {"@tc": "21474836", "#text": "FINE"}, "Teacher": [ {"@id": "XYZ_1", "TeacherType": {"@tc": "5", "#text": "GENDER"}, "Gender": "FEMALE", "Extension": {"@VC": "23", "Extension_Teacher": {"DateDuration": {"@tc": "10023111", "#text": "0-6 MONTHS"}}}}, {"@id": "XYZ_2", "TeacherType": {"@tc": "23", "#text": "EDUCATED"}, "Extension": {"@VC": "23", "Extension_Teacher": {"DateDuration": {"@tc": "10023111", "#text": "CURRENT"}}}}]}, {"@id": "ABC_4", "SchoolType": {"@tc": "21474836", "#text": "OTHER DAYS"}, "SchoolStatus": {"@tc": "1", "#text": "DOING OKAY"}, "Extension": {"Extension_School": {"AdditionalDetails": "CHRISTMAS DAY"}}}] }  I want to extract the Teacher information (TeacherType, Gender etc.) for each Teacher @id where the associated SchoolType.\"#text\" contains "OTHER" for any School @id for all School. I tried the below query but it doesn't work: School[?SchoolType.\"#text\".contains(@, "OTHER")].Teacher[*].TeacherType.\"#text\"[]] 

Answer: You have to warp the contains function around the array of your condition this way: [?contains(SchoolType."#text", 'OTHER')]  So a way to get the full Teacher object would be: School[?contains(SchoolType."#text", 'OTHER')].Teacher  Or, getting rid of the array of array with a flatten operator: School[?contains(SchoolType."#text", 'OTHER')].Teacher | []  This would give: [ { "@id": "XYZ_1", "TeacherType": { "@tc": "5", "#text": "GENDER" }, "Gender": "FEMALE", "Extension": { "@VC": "23", "Extension_Teacher": { "DateDuration": { "@tc": "10023111", "#text": "0-6 MONTHS" } } } }, { "@id": "XYZ_2", "TeacherType": { "@tc": "23", "#text": "EDUCATED" }, "Extension": { "@VC": "23", "Extension_Teacher": { "DateDuration": { "@tc": "10023111", "#text": "CURRENT" } } } } ]
Question: So I am aware of virtualenv in PowerShell? question, specificly this answer, stating that all you need to activate is to execute venv/Scripts/activate.ps1  Of course it is required to set appropriate Execution Policy etc. But my question is: How to disable virtualenv activated inside Power Shell? I tried: venv/Scripts/deactivate.ps1  and venv/Scripts/activate.ps1 deactivate  but first one fails because deactivate.ps1 does not exists and second one does not change anything.

Answer: Just a moment after creating this question I've realized that answer is much simpler that I expected. All I need is to type deactivate  inside virtualenv.
Question: I am working with GeoPandas and I have two GeoDataframes with the same CRS. One of them contains a geometry column with a polygon geometry, the other one a column with point geometry. I want to check which points are inside the polygon. Naively I tried shape.contains(points)  This gave me > The indices of the two GeoSeries are different  I do not understand this message. When I check the documentation, it says  We can also check two GeoSeries against each other, row by row. The GeoSeries above have different indices. We can either align both GeoSeries based on index values and compare elements with the same index using align=True or ignore index and compare elements based on their matching order using align=False:  What are these Indices? Why are they checked against each other and not the geometry columns? Online I read, I have to convert my geometries into shapely geometries. But isn't the whole point of using GeoPandas that I can perfrom geographical operations on the data? I am confused about this. How to check if the geometries in shape contain any of the geometries in points?

Answer: What you are describing is effectively a spatial join. Below example constructs points from lon/lat of cities in UK and then finds which administrational area polygon the city is in. This is an NxM comparison import pandas as pd import numpy as np import geopandas as gpd import shapely.geometry import requests # source some points and polygons # fmt: off dfp = pd.read_html("https://www.latlong.net/category/cities-235-15.html")[0] dfp = gpd.GeoDataFrame(dfp, geometry=dfp.loc[:,["Longitude", "Latitude",]].apply(shapely.geometry.Point, axis=1)) res = requests.get("https://opendata.arcgis.com/datasets/69dc11c7386943b4ad8893c45648b1e1_0.geojson") df_poly = gpd.GeoDataFrame.from_features(res.json()) # fmt: on gpd.sjoin(dfp, df_poly)
Question: If a DataFrame is constructed like the following using polars-python: import polars as pl from polars import col from datetime import datetime df = pl.DataFrame({ "dates": ["2016-07-02", "2016-08-10", "2016-08-31", "2016-09-10"], "values": [1, 2, 3, 4] })  How to select the rows between a certain date range, i.e. between between "2016-08-10" and "2016-08-31", so that the desired outcome is: ┌────────────┬────────┐ │ dates ┆ values │ │ --- ┆ --- │ │ date ┆ i64 │ ╞════════════╪════════╡ │ 2016-08-10 ┆ 2 │ │ 2016-08-31 ┆ 3 │ └────────────┴────────┘ 

Answer: First you need transform the string values to date types then filter: # eager (df.with_columns(pl.col("dates").str.to_date()) .filter(col("dates").is_between(datetime(2016, 8, 9), datetime(2016, 9, 1))) ) # lazy (df.lazy() .with_columns(pl.col("dates").str.to_date()) .filter(col("dates").is_between(datetime(2016, 8, 9), datetime(2016, 9, 1))) .collect() )  both result in the desired output: ┌────────────┬────────┐ │ dates ┆ values │ │ --- ┆ --- │ │ date ┆ i64 │ ╞════════════╪════════╡ │ 2016-08-10 ┆ 2 │ │ 2016-08-31 ┆ 3 │ └────────────┴────────┘
Question: I am breaking my head trying to figure out how to use group_by and apply a custom function using Polars. Coming from Pandas, I was using: import pandas as pd from scipy.stats import spearmanr def get_score(df): return spearmanr(df["prediction"], df["target"]).correlation df = pd.DataFrame({ "era": [1, 1, 1, 2, 2, 2, 5], "prediction": [2, 4, 5, 190, 1, 4, 1], "target": [1, 3, 2, 1, 43, 3, 1] }) correlations = df.groupby("era").apply(get_score)  Polars has map_groups() to apply a custom function over groups, which I tried: correlations = df.group_by("era").map_groups(get_score)  But this fails with the error message:  'Could not get DataFrame attribute '_df'. Make sure that you return a DataFrame object.: PyErr { type: <class 'AttributeError'>, value: AttributeError("'float' object has no attribute '_df'"), traceback: None }  Any ideas?

Answer: Polars has the pl.corr() function which supports method="spearman" If you want to use a custom function you could do it like this: Custom function on multiple columns/expressions import polars as pl from typing import List from scipy import stats df = pl.DataFrame({ "g": [1, 1, 1, 2, 2, 2, 5], "a": [2, 4, 5, 190, 1, 4, 1], "b": [1, 3, 2, 1, 43, 3, 1] }) def get_score(args: List[pl.Series]) -> pl.Series: return pl.Series([stats.spearmanr(args[0], args[1]).correlation], dtype=pl.Float64) (df.group_by("g", maintain_order=True) .agg( pl.map_groups( exprs=["a", "b"], function=get_score).alias("corr") ))  Polars provided function (df.group_by("g", maintain_order=True) .agg( pl.corr("a", "b", method="spearman").alias("corr") ))  Both output: shape: (3, 2) ┌─────┬──────┐ │ g ┆ corr │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═════╪══════╡ │ 1 ┆ 0.5 │ │ 2 ┆ -1.0 │ │ 5 ┆ NaN │ └─────┴──────┘  Custom function on a a single column/expression We can also apply custom functions on single expressions, via .map_elements Below is an example of how we can square a column with a custom function and with normal polars expressions. The expression syntax should always be preferred, as its a lot faster. (df.group_by("g") .agg( pl.col("a").map_elements(lambda group: group**2).alias("squared1"), (pl.col("a")**2).alias("squared2") ))
Question: I'm trying to run an example from the docs, but get the error: Traceback (most recent call last): File "<stdin>", line 2, in <module> TypeError: Point() accepts 0 positional sub-patterns (2 given)  Can someone explain what I doing wrong here? class Point(): def __init__(self, x, y): self.x = x self.y = y x, y = 5 ,5 point = Point(x, y) match point: case Point(x, y) if x == y: print(f"Y=X at {x}") case Point(x, y): print(f"Not on the diagonal") 

Answer: You need to define __match_args__ in your class. As pointed at in this section of the "What's new in 3.10" page:  You can use positional parameters with some builtin classes that provide an ordering for their attributes (e.g. dataclasses). You can also define a specific position for attributes in patterns by setting the __match_args__ special attribute in your classes. If it’s set to (“x”, “y”), the following patterns are all equivalent (and all bind the y attribute to the var variable): Point(1, var) Point(1, y=var) Point(x=1, y=var) Point(y=var, x=1)   So your class will need to look as follows: class Point: __match_args__ = ("x", "y") def __init__(self, x, y): self.x = x self.y = y  Alternatively, you could change your match structure to the following: match point: case Point(x=x, y=y) if x == y: print(f"Y=X at {x}") case Point(x=x, y=y): print(f"Not on the diagonal")  (Note that you don't need a both: a class with __match_args__ defined, does not need to have its arguments specified in the match-case statements.) For full details, I'll refer readers to PEP 634, which is the specification for structural pattern matching. The details on this particular point are in the section Class Patterns. For a better introduction or tutorial, don't use the "What's New" documentation, as it tends to provide an overview, but may skip over some things. Instead, use PEP 636 -- Structural Pattern Matching: Tutorial, or the language reference on match statements for more details.  It is mentioned in the quoted text that a dataclass will already have an ordering, and in your example, a dataclass also works fine: from dataclasses import dataclass @dataclass class Point: x: int y: int x, y = 5, 5 point = Point(x, y) match point: case Point(x, y) if x == y: print(f"Y=X at {x}") case Point(x, y): print(f"Not on the diagonal")
Question: I just bought a new laptop and I'm trying to set it up with python. I am using python 3.10.0, windows 10, pip v21.3. For the most part, pip seems to be working correctly, I've already used it to install multiple packages such as pygame. When I try to install winrt, however, I get this error C:\Users\matth>pip install winrt ERROR: Could not find a version that satisfies the requirement winrt (from versions: none) ERROR: No matching distribution found for winrt  My old laptop is still able to uninstall and reinstall winrt using pip without a problem, and again pip works on my new laptop for other packages, just not winrt. Any idea what the problem is and how I fix it?

Answer: Microsoft has not been maintaining the winrt package. There is no binary wheel for Python 3.10 as seen on PyPI. There is also a request for this on GitHub. I have started a community-maintained fork of the PyWinRT project. You can install and use winsdk instead. It supports Python 3.10. Just replace winrt with winsdk in your imports. EDIT: Since September 2023, the monolithic winsdk package was getting too big, so it has been replaced by individual winrt-* packages for each WinRT namespace. winrt-runtime is the base shared runtime support package and each namespace has a package similar to winrt-Windows.Foundation. Documentation can be found here. A list of improvements and bug fixes compared to the winrt package can be found in the changelog.
Question: I have defined a pydantic Schema with extra = Extra.allow in Pydantic Config. Is it possible to get a list or set of extra fields passed to the Schema separately. For ex: from pydantic import BaseModel as pydanticBaseModel class BaseModel(pydanticBaseModel): name: str class Config: allow_population_by_field_name = True extra = Extra.allow  I pass below JSON:  { "name": "Name", "address": "bla bla", "post": "post" }  I need a function from pydantic, if available which will return all extra fields passed. like: {"address", "post"}

Answer: NOTE: This answer applies to version Pydantic 1.x. See robcxyz's answer for a 2.x solution.  From the pydantic docs:  extra whether to ignore, allow, or forbid extra attributes during model initialization. Accepts the string values of 'ignore', 'allow', or 'forbid', or values of the Extra enum (default: Extra.ignore). 'forbid' will cause validation to fail if extra attributes are included, 'ignore' will silently ignore any extra attributes, and 'allow' will assign the attributes to the model.  This can either be included in the model Config class, or as arguments when inheriting BaseModel. from pydantic import BaseModel, Extra class BaseModel(BaseModel, extra=Extra.allow): name: str model = Model.parse_obj( {"name": "Name", "address": "bla bla", "post": "post"} ) print(model) # name='Name' post='post' address='bla bla'  To get the extra values you could do something simple, like comparing the set of __fields__ defined on the class to the values in __dict__ on an instance: class Model(BaseModel, extra=Extra.allow): python_name: str = Field(alias="name") @property def extra_fields(self) -> set[str]: return set(self.__dict__) - set(self.__fields__)  >>> Model.parse_obj({"name": "Name", "address": "bla bla", "post": "post"}).extra_fields {'address', 'post'} >>> Model.parse_obj({"name": "Name", "foobar": "fizz"}).extra_fields {'foobar'} >>> Model.parse_obj({"name": "Name"}).extra_fields set()