eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In Pydantic I want to represent a list of items as a dictionary. In that dictionary I want the key to be the id of the Item to be the key of the dictionary. I read the documentation on Serialization, on Mapping types and on Sequences. However, I didn't find a way to create such a representation. I want these in my api to have easy access from generated api clients. class Item(BaseModel): uid: UUID = Field(default_factory=uuid4) updated: datetime = Field(default_factory=datetime_now) ... class ResponseModel: ... print ResponseModel.model_dump() #> { "67C812D7-B039-433C-B925-CA21A1FBDB23": { "uid": "67C812D7-B039-433C-B925-CA21A1FBDB23", "updated": "2024-05-02 20:24:00" },{ "D39A8EF1-E520-4946-A818-9FA8664F63F6": { "uid": "D39A8EF1-E520-4946-A818-9FA8664F63F6", "updated":"2024-05-02 20:25:00" } } 

Answer: You can use PlainSerializer(docs) to change the serialization of a field, e.g., to change the desired output format of datetime: from pydantic import Field, BaseModel, PlainSerializer from uuid import UUID, uuid4 from datetime import datetime from typing import Annotated CustomUUID = Annotated[ UUID, PlainSerializer(lambda v: str(v), return_type=str) ] CustomDatetime = Annotated[ datetime, PlainSerializer( lambda v: v.strftime("%Y-%m-%d %H:%M:%S"), return_type=str ), ] class Item(BaseModel): uid: CustomUUID = Field(default_factory=uuid4) updated: CustomDatetime = Field(default_factory=datetime.now) Item().model_dump() # {'uid': '7b2b8e32-15c7-48b8-9ddb-2d25cc61bc61', # 'updated': '2024-05-03 00:13:38'}  Then you can use model_serializer (docs) to serialize the ResponseModel: from pydantic import Field, BaseModel, model_serializer, PlainSerializer from uuid import UUID, uuid4 from datetime import datetime from typing import Annotated CustomUUID = Annotated[ UUID, PlainSerializer(lambda v: str(v), return_type=str) ] CustomDatetime = Annotated[ datetime, PlainSerializer( lambda v: v.strftime("%Y-%m-%d %H:%M:%S"), return_type=str ), ] class Item(BaseModel): uid: CustomUUID = Field(default_factory=uuid4) updated: CustomDatetime = Field(default_factory=datetime.now) class ResponseModel(BaseModel): items: list[Item] @model_serializer def serialize_model(self): return {str(item.uid): item.model_dump() for item in self.items} ResponseModel(items=[Item() for i in range(2)]).model_dump() # { # "e41c4446-60e3-45c5-ab0a-a25f15febac4": { # "uid": "e41c4446-60e3-45c5-ab0a-a25f15febac4", # "updated": "2024-05-03 00:14:58", # }, # "e07d4152-755a-4a0f-9312-29d343bda883": { # "uid": "e07d4152-755a-4a0f-9312-29d343bda883", # "updated": "2024-05-03 00:14:58", # }, # }
Question: I'm using the ijson library in Python to parse a large JSON file and I need to find the position in the file where specific data is located. I want to use file.tell() to get the current position of the file reader during the parsing process. But it's only giving me the length of the file. from ijson import parse with open('file','r') as f: for a, b, c in parse(f): print(f.tell()) 

Answer: ijson.parse is using buffered reads of the source file: >>> help(ijson.parse) Help on function parse in module ijson.common: parse(source, buf_size=65536, **config)  If your file is smaller than 64K it will look like f.tell() is returning the file size. If you use parse(f, buf_size=1) the f.tell() should be accurate, but parsing will likely be slower.
Question: I want to overload the function below so if passed a value that supports int() Python type hints int otherwise Python type hints the value passed. Python typing module provides a SupportsInt type we can use to check if our value supports int. from typing import Any, SupportsInt, overload @overload def to_int(value: SupportsInt) -> int: ... @overload def to_int[T: NotSupportsInt???](value: T) -> T: ... def to_int(value: Any) -> Any: try: return int(value) except TypeError: return value  But in our second overload statement how can we specify all values that don't support int?

Answer: Types make positive promises - they specify things an object can do. Not things an object can't do. If you have an object of static type float, you know this object supports __int__, so you know it's a member of type SupportsInt. If you have an object of static type object, object does not have an __int__ method, but this object still might support __int__. The object could be a float, or an instance of some other subclass of object that supports __int__. You don't know whether passing it to to_int will return an int. Your second overload is trying to say "Doesn't support __int__? Then return the original type." What it should say is "Don't know if it supports __int__? Then maybe return the original type, or maybe return int." You'd do that with an unbounded type variable and a union return type, like this: from typing import overload, SupportsInt, TypeVar T = TypeVar('T') @overload def to_int(value: SupportsInt) -> int: ... @overload def to_int(value: str) -> int: ... @overload def to_int(value: T) -> T | int: ... def to_int(value): try: return int(value) except TypeError: return value  Note that I've also added a separate overload for str. str doesn't actually support __int__, so it doesn't fall under the SupportsInt overload - calls like int('35') are handled by a special case in int.__new__. And sure, sometimes you'll have a case like to_int([]), where you know the specific concrete type of the object, and you know the int call will fail. But the type system doesn't propagate that information the way it would need to for more specific annotations to work. To the type checker, an instance of list might be an instance of some weird __int__-supporting list subclass, even if a human looking at the code can see it's not.
Question: Here is a small application that reproduces my problem: import fastapi import logging import loguru instance = fastapi.FastAPI() @instance.on_event("startup") async def startup_event(): logger = logging.getLogger("mylogger") logger.info("I expect this to log") loguru.logger.info("but only this logs")  When I launch this application with uvicorn app.main:instance --log-level=debug I see this in my terminal: INFO: Waiting for application startup. 2024-05-02 13:14:45.118 | INFO | app.main:startup_event:28 - but only this logs INFO: Application startup complete.  Why does only the loguru logline work, and how can I make standard python logging work as expected?

Answer: It's because the --log-level=debug only applies to uvicorn's logger, not your mylogger logger - whose level remains set at WARNING. If you add a line to the end of your script such as logging.basicConfig(level=logging.DEBUG, format="%(asctime)s | %(levelname)-8s | " "%(module)s:%(funcName)s:%(lineno)d - %(message)s")  and run, you'll see something like $ uvicorn main:instance --log-level=debug INFO: Started server process [1311] INFO: Waiting for application startup. 2024-05-02 16:44:55,736 | INFO | main:startup_event:12 - I expect this to log 2024-05-02 16:44:55.736 | INFO | main:startup_event:13 - but only this logs INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)  The extra line configures the root logger and adds a console handler with that format, and events logged to the mylogger logger are passed up the hierarchy to the root logger's console handler.
Question: Sample code: import click @click.command @click.option('-f/-F', 'var', default=True) def main(var): click.echo(var) main()  Inside main() function, how can I check if var parameter got True value by default, or it was passed by the user? What I want to achieve: I will have few flags. When the user does not pass any of the flags, I want them all to be True. When the user passes at least one of the flags, only the passed flags should be True, the other flags False.

Answer: As I haven't really used click - maybe there's some functionality to handle this but as a workaround, I'd set the default value to None and check against if the value if its None when command is called. Something like this: import click @click.command @click.option('-f/-F', 'var', default=None) def main(var): if var is None: var = True print("No Value Provided - default to True") click.echo(var) main()
Question: I am trying to write python code that takes the first of the next month and creates a dataframe of the next 12 months. I have tried two ways to which did not work.  I created a FOR LOOP but I am stuck on the fact that it does not store the range in the list. I got a Value Error: Month must be in 1..12 so I'm stuck as well. Can I get help with either?  # FIRST TRY # import packages # python import pandas as pd import numpy as np import datetime as dt from datetime import timedelta from datetime import datetime from datetime import date from dateutil.relativedelta import relativedelta #get todays date today = dt.date.today() # Get the first day of next month first_day_next_month = (today.replace(day=1) + dt.timedelta(days=32)).replace(day=1) # Get the first day of the next 12 months first_day_next_12_months = [] for i in range(1, 13): next_month = first_day_next_month + relativedelta(months=i) first_day_next_12_months.append(next_month) print(first_day_next_12_months)  #SECOND TRY # import packages # python import pandas as pd import numpy as np import datetime as dt from datetime import timedelta from datetime import datetime from datetime import date from dateutil.relativedelta import relativedelta #get todays date today = dt.date.today() # Get the first day of next month first_day_next_month = (today.replace(day=1) + dt.timedelta(days=32)).replace(day=1) # Get the first day of next month first_day_next_month = (today.replace(day=1) + dt.timedelta(days=32)).replace(day=1) print("First day of next month:", first_day_next_month) # Get the first day of the next 12 months first_day_next_12_months = [] for i in range(1, 13): next_month = (first_day_next_month.replace(month=first_day_next_month.month + i-1) + datetime.timedelta(days=32)).replace(day=1) first_day_next_12_months.append(next_month) print("First day of the next 12 months:") for date in first_day_next_12_months: print(date) 

Answer: If working with pandas.DataFrame, you mostly don't need for loops. Just do : import pandas as pd import datetime as dt from dateutil.relativedelta import relativedelta today = dt.date.today() first_day_next_month = (today.replace(day=1) + relativedelta(months=1)) first_day_next_12_months = [first_day_next_month + relativedelta(months=i) for i in range(12)] df_months = pd.DataFrame(first_day_next_12_months, columns=['First Day of Month'])
Question: I have jsonl files partitioned by user_id, and report_date. I am converting these jsonl files into parquet files and save them in the same folder using the following commands in DuckDB jsonl_file_path ='/users/user_id=123/report_date=2024-04-30/data.jsonl' out_path = '/users/user_id=123/report_date=2024-04-30/data.parquet' db.sql( f""" COPY ( SELECT * FROM read_json_auto( '{jsonl_file_path}', maximum_depth=-1, sample_size=-1, ignore_errors=true ) ) TO '{out_path}' ( FORMAT PARQUET, ROW_GROUP_SIZE 100000, OVERWRITE_OR_IGNORE 1 ); """ )  It works fine, but the problem is DuckDB is inserting the hive partition values into the parquet file which are user_id and report_date, these values are not in jsonl file. anyone know how to solve this issue?

Answer: If I understand correctly, you want to do a partitioned write and need to use PARTITION_BY. When doing a partitioned write, you should not include the hive partition as part of your output path. The partitioned write will build those paths and file names for the case where there are many files per partition. You can template the file format using FILENAME_PATTERN. jsonl_file_path ='/users/user_id=123/report_date=2024-04-30/data.jsonl' out_path = '/users' file_pattern = 'data_{i}' db.sql( f""" COPY ( SELECT * FROM read_json_auto( '{jsonl_file_path}', maximum_depth=-1, sample_size=-1, ignore_errors=true ) ) TO '{out_path}' ( FORMAT PARQUET, PARTITION_BY (user_id, report_date), FILENAME_PATTERN {file_pattern} ROW_GROUP_SIZE 100000, OVERWRITE_OR_IGNORE 1 ); """ )  If you want to overwrite the single file, modify the projection in the SELECT not to use *. The * includes the "virtual" columns that are part of the hive table's partition. They are columns in the hive table stored in the path, not the physical parquet files. jsonl_file_path ='/users/user_id=123/report_date=2024-04-30/data.jsonl' out_path = '/users/user_id=123/report_date=2024-04-30/data.parquet' db.sql( f""" COPY ( SELECT * EXCLUDE(user_id, report_date) FROM read_json_auto( '{jsonl_file_path}', maximum_depth=-1, sample_size=-1, ignore_errors=true ) ) TO '{out_path}' ( FORMAT PARQUET, ROW_GROUP_SIZE 100000, OVERWRITE_OR_IGNORE 1 ); """ )
Question: Is it possible to reverse an array using as_strided in NumPy? I tried the following and got garbage results Note : I am aware of indexing tricks like ::-1, but what to know if this can be achieved thru ax_strided. import numpy as np from numpy.lib.stride_tricks import as_strided elems = 16 inp_arr = np.arange(elems).astype(np.int8) print(inp_arr) print(inp_arr.shape) print(inp_arr.strides) expanded_input = as_strided(inp_arr[15], shape = inp_arr.shape, strides = (-1,)) print(expanded_input) print(expanded_input.shape) print(expanded_input.strides)  Output [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15] (16,) (1,) [ 15 -113 0 21 -4 -119 -53 3 79 0 0 0 0 0 0 2] (16,) (-1,)

Answer: Indeed you can reverse an array using as_strided() (notwithstanding the question of whether you should); however, one has to get both the strides and the array offset correct:  In the 1-dimensional case, one has to  start from the last element and negate the stride (which, for an 8-bit/1-byte data type, is the same as setting it to -1).   Generally, in the N-dimensional case of an arbitrary data type, one has to  start from the last element in the axis of interest and negate the stride in the axis of interest.    See code below, where r_1d and r_nd are exemplary results of reversing with as_strided(): import numpy as np from numpy.lib.stride_tricks import as_strided # 1D, 8-bit/1-byte dtype: (1) start from last element, (2) set stride == -1 a_1d = np.arange(16, dtype=np.int8) r_1d = as_strided(a_1d[-1:], shape=a_1d.shape, strides=(-1, )) assert (r_1d == a_1d[::-1]).all() # ND, arbitrary dtype: (1) start from last element in axis of interest, (2) # negate stride in axis of interest (below, axis of interest is axis 1) a_nd = np.arange(2*3*5, dtype=float).reshape(2, 3, 5) a_offset = a_nd[:, -1:] r_strides = np.multiply([1, -1, 1], a_nd.strides) r_nd = as_strided(a_offset, shape=a_nd.shape, strides=r_strides) assert (r_nd == a_nd[:, ::-1]).all()  As a cautionary note, I should add that I only tried with contiguous arrays (both C and F), others might require more effort.
Question: In a big library, I had a 'bug' because a function which accept only float but not int def foo(penalty: float): if not isinstance(penalty, float): raise ValueError(f"`penalty` has to be a float, but is {penalty}") # some instruction  I would like to open a PR with the following : def foo(penalty: int | float): if not isinstance(penalty, (int,float)): raise ValueError(f"`penalty` has to be a float or int, but is {penalty}") # some instruction  but before this, I ask myself : are there good reason, in Python to accept float but not int ?

Answer: A few reasons might exist:  Floats have methods that integers don't. For example the fromhex or hex methods exist on float objects, but not int objects. If the function makes use of such methods, accepting an int will cause it to fail.  Sometimes, floats are specifically needed when dealing with foreign functions, for example, a function expecting a C double, a float works for this purpose, but an int does not.  Other types are accepted by isinstance checks of int, for example bool objects, which may be undesirable.  In Python2, arithmetic with integers can sometimes produce different results than when a float is used. If the library is (or was) backwards compatible to Python2, it could be an artifact of that fact  A function may explicitly expect a value that can only be expressed as a float. For example, if a function is expecting a value between (non-inclusive) 0 and 1, an int will never make sense.   Presumably, a function could just upcast the integer to a float as-needed: if isinstance(arg, int): arg = float(arg)  But for performance, it may be better to error out and inform the user a float is needed (so perhaps the user can use a float to begin with).
Question: I need to do this action once under certain conditions in a for loop to stop duplicates appearing in the checkboxes. Theoretically, this should work I believe but when the conditions are met for this if statement (cycles == 1 & len(tasks) > 1) it skips over the statement anyway which really confuses me. I have been trying to debug this with breakpoints but I can't figure it out. tasks = [a, b] def addToList(): cycles = 0 for x in range(len(tasks)): cycles += 1 if cycles == 1 & len(tasks) > 1: checkList.destroy() checkList = Checkbutton(viewTab,text=tasks[x]) checkList.pack(anchor='w',side=BOTTOM) 

Answer: The issue lies in the way you're using the '&' operator. In Python '&' is a bitwise AND operator, not a logical AND operator. For logical AND, you should use 'and' So, instead of: if cycles == 1 & len(tasks) > 1: You should use: if cycles == 1 and len(tasks) > 1:
Question: I currently have code that looks like this scattered throughout my codebase: try: something() except Exception as exc: raise SomethingError from exc  I would like to write a context manager that would remove some of this boiler-plate: with ExceptionWrapper(SomethingError): something()  It looks like it is possible to suppress exceptions inside a context manager - see: contextlib.suprress. It doesn't look like it is possible to change what exception is being raised. However, I haven't been able to find clear documentation on what the return value of the __exit__ function of a context manager is.

Answer: This sort of simple context manager is easiest to implement using contextlib.contextmanager, which creates a context manager out of a generator. Simply wrap the first yield statement in the generator with a try block and raise the desired exception in the except block: import contextlib from typing import Iterator class SomethingError(Exception): pass @contextlib.contextmanager def exception_wrapper(message: str) -> Iterator[None]: """Wrap Exceptions with SomethingError.""" try: yield except Exception as exc: raise SomethingError(message) from exc  This can be used like so: >>> with exception_wrapper("and eggs!"): ... raise Exception("spam") Traceback (most recent call last): File "/path/to/script.py", line 11, in exception_wrapper yield File "/path/to/script.py", line 17, in <module> raise Exception("spam") Exception: spam The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/path/to/script.py", line 16, in <module> with exception_wrapper("and eggs!"): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/path/to/script.py", line 13, in exception_wrapper raise SomethingError(message) from exc __main__.SomethingError: and eggs!
Question: pandera: 0.18.3 pandas: 2.2.2 python: 3.9/3.11  Hi, I am unable to setup the pandera for pandas dataframe as it complains: File "/anaconda/envs/data_quality_env/lib/python3.9/site-packages/pandera/api/base/schema.py", line 96, in get_backend raise BackendNotFoundError( pandera.errors.BackendNotFoundError: Backend not found for backend, class: (<class 'data_validation.schemas.case.CaseSchema'>, <class 'pandas.core.frame.DataFrame'>). Looked up the following base classes: (<class 'pandas.core.frame.DataFrame'>, <class 'pandas.core.generic.NDFrame'>, <class 'pandas.core.base.PandasObject'>, <class 'pandas.core.accessor.DirNamesMixin'>, <class 'pandas.core.indexing.IndexingMixin'>, <class 'pandas.core.arraylike.OpsMixin'>, <class 'object'>)  My folder structure is: project/ data_validation/ schema/ case.py validation/ validations.py pipeline.py  case.py: import pandas as pd import pandera as pa class CaseSchema(pa.DataFrameSchema): case_id = pa.Column(pa.Int)  validations.py import pandas as pd from data_validation.schemas.case import CaseSchema def validate_case_data(df: pd.DataFrame) -> pd.DataFrame: """Validate a DataFrame against the PersonSchema.""" schema = CaseSchema() return schema.validate(df)  pipeline.py import pandas as pd from data_validation.validation.validations import validate_case_data def validate_df(df: pd.DataFrame) -> pd.DataFrame: """Process data, validating it against the PersonSchema.""" validated_df = validate_case_data(df) return validated_df df = pd.DataFrame({ "case_id": [1, 2, 3] }) processed_df = validate_df(df) 

Answer: This can be solved by including a get_backend method in CaseSchema: import pandas as pd import pandera as pa from pandera.backends.pandas.container import DataFrameSchemaBackend class CaseSchema(pa.DataFrameSchema): case_id = pa.Column(pa.Int) @classmethod def get_backend(cls, check_obj=None, check_type=None): if check_obj is not None: check_obj_cls = type(check_obj) elif check_type is not None: check_obj_cls = check_type else: raise ValueError("Must pass in one of `check_obj` or `check_type`.") cls.register_default_backends(check_obj_cls) return DataFrameSchemaBackend()