eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm investigating the pattern whereby you have a method decorator which annotates the method in some way, and then once the class is defined it looks through its methods, finds the annotated methods, and registers or processes them in some way. e.g. def class_decorator(cls): for name, method in cls.__dict__.iteritems(): if hasattr(method, "use_class"): # do something with the method and class print name, cls return cls def method_decorator(view): # mark the method as something that requires view's class view.use_class = True return view @class_decorator class ModelA(object): @method_decorator def a_method(self): # do some stuff pass  (From here: https://stackoverflow.com/a/2367605/1256529) I wondered if anyone has a solution to the problem of multiple decorators. For example, this example works fine in isolation, but if I do the following, it will break because the use_class annotation will be hidden. def hiding_decorator(fn): def wrapper(*args, **kwargs): print("do some logging etc.") return fn(*args, **kwargs) return wrapper @class_decorator class ModelA(object): @hiding_decorator @method_decorator def a_method(self): # do some stuff pass  Does anyone have a reliable and user-friendly way around this problem? Non-optimal solutions I can think of:  Dictate that your decorator should be outermost (but what if other decorators also have this requirement?) Dictate that it only works with decorators that are transparent proxies for attributes of their inner functions/methods. Require other decorators to be aware of the requirements of the tagging decorator (means generic utility decorators become difficult to use).  None of these are particularly attractive.

Answer: Since a "tag" is always applied on a function object itself, it is always going to be susceptible to obstruction by an unaware wrapper function. To achieve a similar usage you can implement the behavior with a registry pattern instead by adding method names to a list with a method decorator so that the class decorator can obtain the final decorated function object with getattr: class Registry: def __init__(self): self.names = [] def register(self, func): self.names.append(func.__name__) return func def apply(self, cls): for name in self.names: print(cls, name) print(getattr(cls, name)) registry = Registry() @registry.apply class ModelA(object): @hiding_decorator @registry.register def a_method(self): pass  This outputs something like: <class '__main__.ModelA'> a_method <function hiding_decorator.<locals>.wrapper at 0x1551795d67a0>  Demo: https://ideone.com/R43Izy
Question: I need -x^{2}+1 rather than 1-x^{2} with sympy.latex(-x**2+1). from sympy import symbols, latex x = symbols('x') print(-x**2+1) print(latex(-x**2+1))  Output: 1 - x**2 1 - x^{2}  Is it possible to change the default format?

Answer: As suggested in comments, you can use the order argument to change the result ordering! https://docs.sympy.org/latest/modules/printing.html#sympy.printing.latex.latex  order: string, optional Any of the supported monomial orderings (currently 'lex', 'grlex', or 'grevlex'), 'old', and 'none'. This parameter does nothing for .Mul objects. Setting order to 'old' uses the compatibility ordering for ~.Add defined in Printer. For very large expressions, set the order keyword to 'none' if speed is a concern.  >>> print(latex(-x**2+1)) 1 - x^{2} >>> print(latex(-x**2+1, order="lex")) - x^{2} + 1  Casually, it could be worth a PR to set this ordering for None when the expression is literally quite short (by count of atoms?), but I'm not sufficiently familiar with real-world cases of it
Question: I have panda series as the following :  1 1 2 2 3 3 4 4 5 0 6 0 7 1 8 2 9 3 10 0 11 0 12 0 13 0 14 1 15 2  I have to arrange this in following format :  1 1 2 2 3 3 4 4 5 0 6 0 7 3 ---> 4-2+1 (previous non zero value - amount of previous zeroes + current value) 8 4 ---> 4-2+2 (previous non zero value - amount of previous zeroes + current value) 9 5 ---> 4-2+3 (previous non zero value - amount of previous zeroes + current value) 10 0 11 0 12 0 13 0 14 2 ---> 5-4+1 (previous non zero value - amount of previous zeroes + current value) 15 3 ---> 5-4+2 (previous non zero value - amount of previous zeroes + current value)  I am stuck at this. Till now I am able to produce a data frame with consecutive zeroes. zero = ser.eq(0).groupby(ser.ne(0).cumsum()).cumsum()  which gave me:  1 0 2 0 3 0 4 0 5 1 6 2 7 0 8 0 9 0 10 1 11 2 12 3 13 4 14 0 15 0  if someone willing to assist on this. i am dropping cookie cutter for this problem which will create the above series. d = {'1': 1, '2': 2, '3': 3, '4':4, '5':0, '6':0, '7':1, '8':2, '9':3, '10':0, '11':0, '12':0, '13':0, '14':1, '15':2} ser = pd.Series(data=d) 

Answer: Although this can only be done with Pandas in a rather convoluted way IMHO, here is a straightforward implementation using Numba (which should also be faster than all Pandas solutions): import numba as nb import numpy as np @nb.njit(['(int32[:],)', '(int64[:],)']) def compute(arr): res = np.empty(arr.size, dtype=arr.dtype) z_count = 0 last_nnz_val = 0 nnz_count = 0 for i in range(arr.size): if arr[i] == 0: if i > 0 and arr[i-1] != 0: # If there is a switch from nnz to zero last_nnz_val += nnz_count - z_count # Save the last nnz result z_count = 0 z_count += 1 res[i] = 0 else: if i > 0 and arr[i-1] == 0: # If there is a switch from zero to nnz nnz_count = 0 nnz_count += 1 res[i] = last_nnz_val - z_count + nnz_count return res # [...] compute(ser.to_numpy())  Note the result is a basic Numpy array, but you can easily create a dataframe from it.  Benchmark Here are performance results on my machine (i5-9600KF CPU) on the tiny example dataset: MichaelCao's answer: 886 µs This answer: 2 µs <-----  On a 1000x larger dataset (repeated), I get: MichaelCao's answer: 1240 µs This answer: 20 µs <-----  It is much faster than the other answer. I also get different output results so one of the answer implementation is certainly wrong.
Question: For a particular JAX function func, one can define non-differentiable arguments by using the decorator @partial(jax.custom_jvp, nondiff_argnums=...). However, in order to make it work, one must also explicitly define the differentiation rules in a custom jvp function by using the decorator @func.defjvp. I'm wondering if there is a generic way to define non-differentiable arguments for any given func, without defining a custom jvp (or vjp) function? This will be useful when the differentiation rules are too complicated to write out.

Answer: In JAX's design, non-differentiated arguments are a property of the gradient transformation being used, not a property of the function being differentiated. custom_jvp is fundamentally about customizing the gradient behavior, and using it to mark non-differentiable arguments without actually customizing the gradient is not an intended use. The way to ensure that arguments do not participate in an autodiff transformation is to specify the arguments you want to differentiate against when you call the jax.grad, jax.jacobian, or other autodiff transformation; e.g. jax.grad(func, argnums=(0,)) # differentiate with respect to argument 0.  Regardless of what func is, this will attempt to differentiate with respect to the 0th argument, and if that argument is either explicitly or implicitly not differentiable due to how func is defined, an error will be raised.
Question: I'm creating a function that yields chunks of an iterable. How can I properly type this function so that the return value bar is of type list[int]. from typing import Any, Generator, Sequence def chunk[T: Any, S: Sequence[T]](sequence: S, size: int) -> Generator[S, None, None]: for i in range(0, len(sequence), size): yield sequence[i : i + size] foo: list[int] = [1,2,3] bar = chunk(foo, 1)  Sequence[T] is invalid because TypeVar constraint type cannot be generic PylancereportGeneralTypeIssues

Answer: The Type Parameter Scopes section of PEP 695 – Type Parameter Syntax specifically points out that Python's typing system currently does not allow a non-concrete upper bound type, but that it may be allowed in a future extension:  # The following generates no compiler error, but a type checker # should generate an error because an upper bound type must be concrete, # and ``Sequence[S]`` is generic. Future extensions to the type system may # eliminate this limitation. class ClassA[S, T: Sequence[S]]: ...   So until that happens you simply have to spell out the generic type in the type annotations instead: def chunk[T](sequence: Sequence[T], size: int) -> Generator[Sequence[T], None, None]: for i in range(0, len(sequence), size): yield sequence[i : i + size]
Question: I have an endpoint that can return JSON or a file (xlsx): class ReportItemSerializer(BaseModel): id: int werks: Annotated[WerkSerializerWithNetwork, Field(validation_alias="werk")] plu: Annotated[str, Field(description="Название PLU")] start_date: Annotated[date, None, Field(description="Дата начала периода в формате")] end_date: Annotated[date | None, Field(description="Дата окончания периода")] anfmenge: Annotated[int | None, Field(des cription="Запас на начало периода")] endmenge: Annotated[int | None, Field(description="Остаток по PLU в разрезе поставщика на конец периода")] soll: Annotated[int, None, Field(description="Поступило за период")] haben: Annotated[ int | None, Field(description="Количество по PLU, которое было возвращено поставщику за указанный период")] model_config = ConfigDict(from_attributes=True) class PaginatedReportItemsSerializer(BaseModel): count: int results: list[ReportItemSerializer] @router.get( "/orders/report", responses={ 200: { "description": "Return report", "content": { "application/json": { "schema": PaginatedReportItemsSerializer.schema(ref_template="#/components/schemas/{model}") }, "application/octet-stream": {} }, }, } ) async def get_report(): pass  With this I have a problem with swagger. With configuration above there is problem:  Nested ReportItemSerializer is not being displayed (string is instead displayed). How can I fix it?

Answer: Here is a working example, heavily based on this answer; hence, please have a look at that answer for more details. The example below will correctly display the example value/schema in Swagger UI autodocs, below the 200 response's description. Using the drop down menu, one could switch between the various media types—in this case, that is application/json and application/octet-stream. Working Example from fastapi import FastAPI from fastapi.openapi.constants import REF_PREFIX from pydantic import BaseModel from datetime import datetime class SubMessage(BaseModel): msg: str dt: datetime = None class Message(BaseModel): msg: str sub: SubMessage def get_200_schema(): return { 'model': Message, 'content': { 'application/json': { 'schema': {'$ref': REF_PREFIX + Message.__name__} }, 'application/octet-stream': { 'schema': {} # whatever } }, } app = FastAPI() @app.get('/', responses={200: get_200_schema()}) async def get_msg(): return Message(msg='main msg', sub=SubMessage(msg='sub msg', dt=datetime.now()))
Question: Best practice for Python is to use venv to isolate to the imports you really need. I use python -m venv. For development, it's very convenient to use IPython and notebooks for exploring code interactively. However, these need to be installed into the venv to be used. That defeats the purpose of the venv. I can make two venvs, one for usage and one venv-ipy for interactive exploration, but that's hard to manage, and need to be synchornized. Is there a solution or practice?

Answer: I work on multiple projects most of which run in production, each with it's own dependencies. The practice which works for me is to create separate requirements.txt files: one for production environment, the other one contains additions used for development, and a small script that creates/u[dates 2 venvs at the same run: <ProjectName>_prod and <ProjectName>_dev. So the requirements_dev.txt can contain only extensions like jupyter, matplotlib and other packages that are used only in development and there is no need to sync the files until reaching the production stage. To install the dependencies in dev environment you mention both files: pip install -r requirements.txt -r requirements_dev.txt  I also use conda to create/manage environments as I find it more convenient than venv.
Question: Here's a file test.xml: <?xml version="1.0" encoding="UTF-8"?> <list> <entry>data</entry> <entry>Łódź</entry> <entry>data Łódź</entry> </list>  and here's a simple python script to parse it into a list with lxml: from lxml import etree class ParseTarget: def __init__(self): self.entries = [] def start(self, tag, attrib): pass def end(self, tag): pass def data(self, data): str = data.strip() if str != '': self.entries.append(data) def close(self): # Reset parser entries = self.entries self.entries = [] # And return results return entries target = etree.XMLParser(target=ParseTarget(), # Including/removing this makes no difference encoding='UTF-8') tree = etree.parse("./test.xml", target) # Expected value of tree: # ['data', 'Łódź', 'data Łódź'] # Actual value of tree # ['data', 'Łódź', 'data ', 'Łódź'] # What gives!!!?  As the comment says, I would expect to end up with a list of three elements, but I get four. This is a minimal demonstration of a general problem: including strings with non-ascii characters (but at least one ascii char at the beginning) results in not a single string, but a list of two strings, split on where the non-ascii chars start. I don't want this to happen (i.e. I want to just get a list of three strings). What should I do? I'm using Python 3.11.2

Answer: You have to use the end handler to reset: Explanation of Steps With event based parsing, the parser may split the third <entry> (<entry>data Łódź</entry>) into multiple data() calls.:  First: "data " (with a space at the end).  Second: "Łódź".   This is why we need to accumulate text correctly to "data Łódź". from lxml import etree class ParseTarget: def __init__(self): self.entries = [] self.current_text = [] def start(self, tag, attrib): self.current_text = [] def end(self, tag): if self.current_text: self.entries.append("".join(self.current_text)) self.current_text = [] # Reset for the next element def data(self, data): if data.strip(): # Ignore completely empty segments but keep spaces self.current_text.append(data) # Append raw data, preserving spaces def close(self): entries = self.entries self.entries = [] return entries target = etree.XMLParser(target=ParseTarget(), encoding='UTF-8') tree = etree.parse("./test.xml", target) print(tree) # ['data', 'Łódź', 'data Łódź']
Question: mypy (v.1.15.0) complains with the following message Access to generic instance variables via class is ambiguous for the following code: from typing import Self class A: B: Self A.B = A()  If I remove B: Self, then is says "type[A]" has no attribute "B". How make mypy happy? You can play with this here: mypy playground

Answer: In addition to InSync's solution, you can also do this: class A: B: 'A' A.B = A()  Or better yet: from typing import ClassVar class A: B: ClassVar['A'] A.B = A()  This is an example of a forward reference, and mypy handles those by putting the type name in quotes. Using ClassVar just tells the type checker to disallow setting B through an instance, since it's supposed to be static. I prefer this over Self because the concept of self in programming usually refers to a specific instance, which doesn't make sense in the context of a static variable.
Question: The code I am using to extract the images is from PIL import Image def extract_images_from_pdfs(pdf_list): import fitz # PyMuPDF output_dir = "C:/path_to_image" os.makedirs(output_dir, exist_ok=True) for pdf_path in pdf_list: pdf_name = os.path.splitext(os.path.basename(pdf_path))[0] # Open the PDF pdf_document = fitz.open(pdf_path) # Track the count of images extracted per page image_count = 0 for page_num, page in enumerate(pdf_document): # Get the images on this page image_list = page.get_images(full=True) if not image_list: print(f"No images found on page {page_num+1} of {pdf_name}") continue # Process each image for img_index, img in enumerate(image_list): xref = img[0] base_image = pdf_document.extract_image(xref) if base_image: image_bytes = base_image["image"] image_ext = base_image["ext"] # Convert bytes to image image = Image.open(io.BytesIO(image_bytes)) # Save the image image_name = f"{pdf_name}_image_{image_count}.{image_ext}" image_path = os.path.join(output_dir, image_name) image.save(image_path) image_count += 1 pdf_document.close() print(f"Extracted {image_count} images from {pdf_name}")  The input, pdf_list, is just a list containing all the names of my pdf's. Extracted image 1  Extracted image 2  Expected image:  Could it be that the images on the PDF are encrypted / accessible and is there a work around for this. Any help is greatly appreciated. testingpdfexampaper.tiiny.site This is the URL for the PDF

Answer: The PDF has 78 very small pieces of imagery of which the "largest" is masking for O on the first page:   1 60 image 81 62 index 1 8 image no 271 0 151 151 1996B 40%  And many are simply one single pixel. They can be in any order and the early ones of the 78 are generally parts of R:   pdfimages -list chem.pdf page num type width height color comp bpc enc interp object ID x-ppi y-ppi size ratio -------------------------------------------------------------------------------------------- 1 0 image 4 26 cmyk 4 8 image no 214 0 163 153 77B 19% 1 1 image 2 2 cmyk 4 8 image no 215 0 204 245 21B 131% 1 2 image 7 59 index 1 8 image no 226 0 306 303 53B 13% 1 3 image 60 39 index 1 8 image no 237 0 150 153 819B 35% 1 4 image 1 1 cmyk 4 8 image no 248 0 204 204 14B 350% 1 5 image 9 4 cmyk 4 8 image no 259 0 162 153 74B 51% 1 6 image 58 31 index 1 8 image no 270 0 150 154 526B 29% 1 7 image 4 3 cmyk 4 8 image no 281 0 153 153 38B 79% 1 8 image 2 2 cmyk 4 8 image no 290 0 153 175 24B 150%  NOTE there is common with many PDF constructions no "one to one" relationship. One text line can be many places and one visible line can be multiple paths too.  Thus image extraction is of no real value as any whole page could be exported as single images, then trimmed to desired areas, at any density/quality you wish.  Python has PyMuPDF which can "gather" "paths" and combine into single graphical units. So if you select an area of inclusions (Region of Interest) they can possibly be reused as vectors elsewhere? This is similar in effect to the way the MuPDF command line can with a few well chosen commands export SVG areas for reuse.
Question: Problem Description I'm experiencing an issue where images uploaded through Django (using boto3) to SeaweedFS's S3 API are corrupted, while uploads through S3 Browser desktop app work correctly. The uploaded files are 55 bytes larger than the original and contain a Content-Encoding: aws-chunked header, making the images unopenable. Environment Setup  Storage: SeaweedFS with S3 API Proxy: nginx (handling SSL) Framework: Django Storage Client: boto3 Upload Method: Using Django's storage backend with PrivateStorage  Issue Details  When uploading through S3 Browser desktop app:  File size matches original Image opens correctly No corruption issues   When uploading through Django/boto3:  File size increases by 55 bytes Response includes Content-Encoding: aws-chunked Image becomes corrupted and unopenable First bytes contain unexpected data (100000) Last bytes end with .x-amz-checksum-    Example of Corrupted File Original file size: 12345 bytes Uploaded file size: 12400 bytes (+55 bytes) First bytes: 100000... Last bytes: ...x-amz-checksum-crc32:SJJ2UA==  Attempted Solutions  Tried different upload methods: # Method 1: Using ContentFile storage.save(path, ContentFile(file_content)) # Method 2: Using Django File object storage.save(path, File(file)) # Method 3: Direct boto3 upload client.upload_fileobj(f, bucket_name, path)    Questions  Is this a known issue with SeaweedFS's S3 API implementation? Is there a way to disable the aws-chunked encoding in boto3? Are there specific headers or configurations needed in the nginx proxy to handle binary uploads correctly?  Additional Information  Django version: 5.1.7 boto3 version: 1.37.23  Any help or insights would be greatly appreciated 🙏!  Sample code I tried: import boto3 AWS_ACCESS_KEY_ID='' AWS_SECRET_ACCESS_KEY='' API_URL='' bucket_name = 'sample-bucket' s3 = boto3.client('s3', aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY, endpoint_url=API_URL) testfile = r"image.png" s3.upload_file(testfile, bucket_name, 'sample.png', ExtraArgs={'ContentType': 'image/png'}) 

Answer: Follow answer in https://github.com/boto/boto3/issues/4435#issuecomment-2648819900 I added this lines on top of settings.py file and problem solved. import os os.environ["AWS_REQUEST_CHECKSUM_CALCULATION"] = "when_required" os.environ["AWS_RESPONSE_CHECKSUM_VALIDATION"] = "when_required"
Question: Question: I am trying to fine-tune the Mistral-7B-Instruct-v0.1-GPTQ model using SFTTrainer from trl. However, when running my script in Google Colab, I encounter the following error: TypeError: SFTTrainer.__init__() got an unexpected keyword argument 'tokenizer'  Here is the relevant portion of my code: import torch from datasets import load_dataset, Dataset from peft import LoraConfig, AutoPeftModelForCausalLM, prepare_model_for_kbit_training, get_peft_model from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig, TrainingArguments from trl import SFTTrainer, SFTConfig import os # Load and preprocess dataset data = load_dataset("tatsu-lab/alpaca", split="train") data_df = data.to_pandas() data_df = data_df[:5000] data_df["text"] = data_df[["input", "instruction", "output"]].apply( lambda x: "###Human: " + x["instruction"] + " " + x["input"] + " ###Assistant: " + x["output"], axis=1 ) data = Dataset.from_pandas(data_df) # Load tokenizer tokenizer = AutoTokenizer.from_pretrained("TheBloke/Mistral-7B-Instruct-v0.1-GPTQ") tokenizer.pad_token = tokenizer.eos_token # Load and prepare model quantization_config_loading = GPTQConfig(bits=4, disable_exllama=True, tokenizer=tokenizer) model = AutoModelForCausalLM.from_pretrained( "TheBloke/Mistral-7B-Instruct-v0.1-GPTQ", device_map="auto" ) model.config.use_cache = False model.config.pretraining_tp = 1 model.gradient_checkpointing_enable() model = prepare_model_for_kbit_training(model) # LoRA configuration peft_config = LoraConfig( r=16, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", target_modules=["q_proj", "v_proj"] ) model = get_peft_model(model, peft_config) # Training configuration training_arguments = SFTConfig( output_dir="mistral-finetuned-alpaca", per_device_train_batch_size=8, gradient_accumulation_steps=1, optim="paged_adamw_32bit", learning_rate=2e-4, lr_scheduler_type="cosine", save_strategy="epoch", logging_steps=100, num_train_epochs=1, max_steps=250, fp16=True, packing=False, max_seq_length=512, dataset_text_field="text", push_to_hub=True ) # Initialize trainer trainer = SFTTrainer( model=model, train_dataset=data, peft_config=peft_config, args=training_arguments, tokenizer=tokenizer, # This causes the error ) trainer.train()  What I Have Tried:  I checked the TRL documentation but couldn't find a tokenizer parameter in the SFTTrainer class.  Removing tokenizer=tokenizer avoids the error, but then I get a different error about tokenization during training.  I tried passing tokenizer inside training_arguments, but that didn't work either.   Question: Is SFTTrainer expecting the tokenizer to be handled differently in the latest versions of trl? How should I correctly pass the tokenizer to ensure training works? I would appreciate any insights!

Answer: In the 0.12.0 release it is explained that the tokenzier argument is now called the processing_class parameter. You should be able to run your code as before by replacing tokenizer with processing_class: trainer = SFTTrainer( model=model, train_dataset=data, peft_config=peft_config, args=training_arguments, processing_class=tokenizer, # This causes the error )