eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using the StableDiffusionPipeline from the Hugging Face Diffusers library in Python 3.10.2, on an M2 Mac (I tagged it because this might be the issue). When I try to generate 1 image from 1 prompt, the output looks fine, but when I try to generate multiple images using the same prompt, the images are all either black squares or a random image (see example below). What could be the issue? My code is as follows (where I change n_imgs from 1 to more than 1 to break it): from diffusers import StableDiffusionPipeline pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5") pipe = pipe.to("mps") # for M1/M2 chips pipe.enable_attention_slicing() prompt = "a photo of an astronaut driving a car on mars" # First-time "warmup" pass (because of weird M1 behaviour) _ = pipe(prompt, num_inference_steps=1) # generate images n_imgs = 1 imgs = pipe([prompt] * n_imgs).images  I also tried setting num_images_per_prompt instead of creating a list of repeated prompts in the pipeline call, but this gave the same bad results. Example output (for multiple images):  [edit/update]: When I generate the images in a loop surrounding the pipe call instead of passing an iterable to the pipe call, it does work: # generate images n_imgs = 3 for i in range(n_imgs): img = pipe(prompt).images[0] # do something with img  But it is still a mystery to me as to why.

Answer: Apparently it is indeed an Apple Silicon (M1/M2) issue, of which Hugging Face is not yet sure why this is happening, see this GitHub issue for more details.
Question: i would like some help. I have multiple excel files, each file only has one sheet. I would like to combine all excel files into just one file but with multiple sheets one sheet per excel file keeping the same sheet names. this is what i have so far: import pandas as pd from glob import glob import os excelWriter = pd.ExcelWriter("multiple_sheets.xlsx",engine='xlsxwriter') for file in glob('*.xlsx'): df = pd.read_excel(file) df.to_excel(excelWriter,sheet_name=file,index=False) excelWriter.save()  All the excel files looks like this: https://iili.io/HfiJRHl.png sorry i cannot upload images here, dont know why but i pasted the link But all the excel files have the exact same columns and rows and just one sheet, the only difference is the sheet name Thanks in advance

Answer: import pandas as pd import os output_excel = r'/home/bera/Desktop/all_excels.xlsx' #List all excel files in folder excel_folder= r'/home/bera/Desktop/GIStest/excelfiles/' excel_files = [os.path.join(root, file) for root, folder, files in os.walk(excel_folder) for file in files if file.endswith(".xlsx")] with pd.ExcelWriter(output_excel) as writer: for excel in excel_files: #For each excel sheet_name = pd.ExcelFile(excel).sheet_names[0] #Find the sheet name df = pd.read_excel(excel) #Create a dataframe df.to_excel(writer, sheet_name=sheet_name, index=False) #Write it to a sheet in the output excel
Question: Suppose that I have this architecture for my classes: # abstracts.py import abc class AbstractReader(metaclass=abc.ABCMeta): @classmethod def get_reader_name(cl): return cls._READER_NAME @classmethod @property @abc.abstractmethod def _READER_NAME(cls): raise NotImplementedError # concretes.py from .abstracts import AbstractReader class ReaderConcreteNumber1(AbstractReader): _READER_NAME = "NAME1" class ReaderConcreteNumber2(AbstractReader): _READER_NAME = "NAME2"  Also I have a manager classes that find concrete classes by _READER_NAME variable. So I need to define unique _READER_NAME for each of my concrete classes. how do I check that NAME1 and NAME2 are unique when concrete classes are going to define?

Answer: You can create a metaclass with a constructor that uses a set to keep track of the name of each instantiating class and raises an exception if a given name already exists in the set: class UniqueName(type): names = set() def __new__(metacls, cls, bases, classdict): name = classdict['_READER_NAME'] if name in metacls.names: raise ValueError(f"Class with name '{name}' already exists.") metacls.names.add(name) return super().__new__(metacls, cls, bases, classdict)  And make it the metaclass of your AbstractReader class. since Python does not allow a class to have multiple metaclasses, you would need to make AbstractReader inherit from abc.ABCMeta instead of having it as a metaclass: class AbstractReader(abc.ABCMeta, metaclass=UniqueName): ... # your original code here  Or if you want to use ABCMeta as metaclass in your AbstractReader, just override ABCMeta class and set child ABC as metaclass in AbstractReader: class BaseABCMeta(abc.ABCMeta): """ Check unique name for _READER_NAME variable """ _readers_name = set() def __new__(mcls, name, bases, namespace, **kwargs): reader_name = namespace['_READER_NAME'] if reader_name in mcls._readers_name: raise ValueError(f"Class with name '{reader_name}' already exists. ") mcls._readers_name.add(reader_name) return super().__new__(mcls, name, bases, namespace, **kwargs) class AbstractReader(metaclass=BaseABCMeta): # Your codes ...  So that: class ReaderConcreteNumber1(AbstractReader): _READER_NAME = "NAME1" class ReaderConcreteNumber2(AbstractReader): _READER_NAME = "NAME1"  would produce: ValueError: Class with name 'NAME1' already exists.  Demo: https://replit.com/@blhsing/MerryEveryInternet
Question: When I recreate and reassign a JAX np array to the same variable name, for some reason the GPU memory nearly doubles the first recreation and then stays stable for subsequent recreations/reassignments. Why does this happen and is this generally expected behavior for JAX arrays? Fully runnable minimal example: https://colab.research.google.com/drive/1piUvyVylRBKm1xb1WsocsSVXJzvn5bdI?usp=sharing. For posterity in case colab goes down: %env XLA_PYTHON_CLIENT_PREALLOCATE=false import jax from jax import numpy as jnp from jax import random # First creation of jnp array x = jnp.ones(shape=(int(1e8),), dtype=float) get_gpu_memory() # the memory usage from the first call is 618 MB # Second creation of jnp array, reassigning it to the same variable name x = jnp.ones(shape=(int(1e8),), dtype=float) get_gpu_memory() # the memory usage is now 1130 MB - almost double! # Third creation of jnp array, reassigning it to the same variable name x = jnp.ones(shape=(int(1e8),), dtype=float) get_gpu_memory() # the memory usage is stable at 1130 MB.  Thank you!

Answer: The reason for this behavior comes from the interaction of several things:  Without pre-allocation, the GPU memory usage will grow as needed, but will not shrink when buffers are deleted.  When you reassign a python variable, the old value still exists in memory until the Python garbage collector notices it is no longer referenced, and deletes it. This will take a small amount of time to occur in the background (you can call import gc; gc.collect() to force this to happen at any point).  JAX sends instructions to the GPU asynchronously, meaning that once Python garbage-collects a GPU-backed value, the Python script may continue running for a short time before the corresponding buffer is actually removed from the device.   All of this means there's some delay between unassigning the previous x value, and that memory being freed on the device, and if you're immediately allocating a new value, the device will likely expand its memory allocation to fit the new array before the old one is deleted. So why does the memory use stay constant on the third call? Well, by this time the first allocation has been removed, and so there is already space for the third allocation without growing the memory footprint. With these things in mind, you can keep the allocation constant by putting a delay between deleting the old value and creating the new value; i.e. replace this: x = jnp.ones(shape=(int(1e8),), dtype=float)  with this: del x time.sleep(1) x = jnp.ones(shape=(int(1e8),), dtype=float)  When I run it this way, I see constant memory usage at 618MiB.
Question: I wish to run a line detector between two known points on an image but firstly I need to widen the area around the line so my line detector has more area to work with. The main issue it stretch the area around line with respect to the line slope. For instance: white line generated form two points with black bounding box. I tried manualy manipulating the box array: input_to_min_area = np.array([[660, 888], [653, 540]]) # this works instead of contour as an input to minAreaRect rect = cv.minAreaRect(input_to_min_area) box = cv.boxPoints(rect) box[[0, 3], 0] += 20 box[[1, 2], 0] -= 20 box = np.int0(box) cv.drawContours(self.images[0], [box], 0, (0, 255, 255), 2)  But that doesn't work for any line slope. From vertical to this angle everything is fine, but for the horizontal lines doesn't work. What would be a simpler solution that works for any line slope?

Answer: A minAreaRect() gives you a center point, the size of the rectangle, and an angle. You could just add to the shorter side length of the rectangle. Then you have a description of a "wider rectangle". You can then do with it whatever you want, such as call boxPoints() on it. padding = 42 rect = cv.minAreaRect(input_to_min_area) (center, (w,h), angle) = rect # take it apart if w < h: # we don't know which side is longer, add to shorter side w += padding else: h += padding rect = (center, (w,h), angle) # rebuild  A box around your two endpoints, widened:
Question: Is there any way to type a Numpy array as generic? I'm currently working with Numpy 1.23.5 and Python 3.10, and I can't type hint the following example. import numpy as np import numpy.typing as npt E = TypeVar("E") # Should be bounded to a numpy type def double_arr(arr: npt.NDArray[E]) -> npt.NDArray[E]: return arr * 2  What I expect arr = np.array([1, 2, 3], dtype=np.int8) double_arr(arr) # npt.NDAarray[np.int8] arr = np.array([1, 2.3, 3], dtype=np.float32) double_arr(arr) # npt.NDAarray[np.float32]  But I end up with the following error arr: npt.NDArray[E] ^^^ Could not specialize type "NDArray[ScalarType@NDArray]" Type "E@double_arr" cannot be assigned to type "generic" "object*" is incompatible with "generic"  If i bound the E to numpy datatypes (np.int8, np.uint8, ...) the type-checker fails to evaluate the multiplication due to the multiple data-types.

Answer: Looking at the source, it seems the generic type variable used to parameterize numpy.dtype of numpy.typing.NDArray is bounded by numpy.generic (and declared covariant). Thus any type argument to NDArray must be a subtype of numpy.generic, whereas your type variable is unbounded. This should work: from typing import TypeVar import numpy as np from numpy.typing import NDArray E = TypeVar("E", bound=np.generic, covariant=True) def double_arr(arr: NDArray[E]) -> NDArray[E]: return arr * 2  But there is another problem, which I believe lies in insufficient numpy stubs. An example of it is showcased in this issue. The overloaded operand (magic) methods like __mul__ somehow mangle the types. I just gave the code a cursory look right now, so I don't know what is missing. But mypy will still complain about the last line in that code:  error: Returning Any from function declared to return "ndarray[Any, dtype[E]]" [no-any-return] error: Unsupported operand types for * ("ndarray[Any, dtype[E]]" and "int") [operator]  The workaround right now is to use the functions instead of the operands (via the dunder methods). In this case using numpy.multiply instead of * solves the issue: from typing import TypeVar import numpy as np from numpy.typing import NDArray E = TypeVar("E", bound=np.generic, covariant=True) def double_arr(arr: NDArray[E]) -> NDArray[E]: return np.multiply(arr, 2) a = np.array([1, 2, 3], dtype=np.int8) reveal_type(double_arr(a))  No more mypy complaints and the type is revealed as follows: numpy.ndarray[Any, numpy.dtype[numpy.signedinteger[numpy._typing._8Bit]]]  It's worth keeping an eye on that operand issue and maybe even report the specific error of Unsupported operand types for * separately. I haven't found that in the issue tracker yet.  PS: Alternatively, you could use the * operator and add a specific type: ignore. That way you'll notice, if/once the annotation error is eventually fixed by numpy because mypy complains about unused ignore-directives in strict mode. def double_arr(arr: NDArray[E]) -> NDArray[E]: return arr * 2 # type: ignore[operator,no-any-return]
Question: I got a warning something like warnings.warn( No local packages or working download links found for tensorflow-text~=2.11.0 error: Could not find suitable distribution for Requirement.parse('tensorflow-text~=2.11.0')  and if I run pip install 'tensorflow-text~=2.11.0' I got : ERROR: Could not find a version that satisfies the requirement tensorflow-text~=2.11.0 (from versions: 2.8.1, 2.8.2, 2.9.0rc0, 2.9.0rc1, 2.9.0, 2.10.0b2, 2.10.0rc0, 2.10.0) ERROR: No matching distribution found for tensorflow-text~=2.11.0  tensorflow-text 2.11.0 available on pypi and if I run pip install tensorflow-text it installs tensorflow-text 2.10.0 and downgrade the whole tensorflow to 2.10.0 Version Info:  OS: Windows 10 Environment: Conda (miniconda3) Python: 3.10.8 Tensorflow: 2.11  I've tried pip and conda-forge

Answer: As per their note, they have dropped building for Windows with v2.11.0. So, you'll need to build from source or seek a third-party build.
Question: . ├── __pycache__ │ └── api.cpython-310.pyc ├── app │ ├── __pycache__ │ │ └── main.cpython-310.pyc │ ├── api_v1 │ │ ├── __pycache__ │ │ │ └── apis.cpython-310.pyc │ │ ├── apis.py │ │ └── endpoints │ │ ├── __pycache__ │ │ │ └── message_prediction.cpython-310.pyc │ │ └── message_prediction.py │ ├── config.py │ ├── main.py │ └── schemas │ ├── Messages.py │ └── __pycache__ │ └── Messages.cpython-310.pyc ├── app.egg-info │ ├── PKG-INFO │ ├── SOURCES.txt │ ├── dependency_links.txt │ └── top_level.txt ├── build │ └── bdist.macosx-12.0-arm64 ├── data │ ├── processed │ │ ├── offers_big.csv_cleaned.xlsx │ │ └── requests_big.csv_cleaned.xlsx │ ├── processed.dvc │ ├── raw │ │ ├── offers.csv.old │ │ ├── offers_big.csv │ │ ├── requests.csv.old │ │ └── requests_big.csv │ ├── raw.dvc │ ├── validated │ │ ├── validated_offers.xlsx │ │ └── validated_requests.xlsx │ └── validated.dvc ├── dist │ └── app-0.1.0-py3.10.egg ├── model.pkl ├── model.py ├── notebooks │ └── contact-form.ipynb ├── requirements.in ├── requirements.txt ├── setup.py └── test_api.py  # main.py import os from fastapi import FastAPI import uvicorn from app.api_v1.apis import api_router # create the app messages_classification_app = FastAPI() messages_classification_app.include_router(api_router) if __name__ == '__main__': uvicorn.run("app.main:messages_classification_app", host=os.getenv("HOST", "0.0.0.0"), port=int(os.getenv("PORT", 8000)))  # requirements.in fastapi uvicorn -e file:.#egg=app  Trying to run the fastAPI app with python, results in error:  py[learning]  ~/r/v/contact-form-classification   master ±  python app/main.py Traceback (most recent call last): File "/Users/xxxxx/repos/visable/contact-form-classification/app/main.py", line 4, in <module> from app.api_v1.apis import api_router ModuleNotFoundError: No module named 'app'  Running it with uvicorn directly works:  py[learning]  ~/r/v/contact-form-classification   master ±  uvicorn app.main:messages_classification_app INFO: Started server process [53665] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)  Any idea why? Looked into similar questions, don't seem to apply to mine.

Answer: python app/main.py will make app/ the first entry in sys.path, so app imports within won't work. Do python -m app.main to run app/main.py as a module without having Python touch sys.path.
Question: I'm trying to write a validator with usage of Pydantic for following strings (examples): 1.1.0, 3.5.6, 1.1.2, etc.. I'm failing with following syntax: install_component_version: constr(regex=r"^[0-9]+.[0-9]+.[0-9]$") install_component_version: constr(regex=r"^([0-9])+.([0-9])+.([0-9])$") install_component_version: constr(regex=r"^([0-9])\.([0-9])\.([0-9])$")  Can anyone help me out what regex syntax should look like?

Answer: The error you are facing is due to type annotation. As per https://github.com/pydantic/pydantic/issues/156 this is not yet fixed, you can try using pydantic.Field and then pass the regex argument there like so install_component_version: str = Field(regex=r"^[0-9]+.[0-9]+.[0-9]$")  This way you get the regex validation and type checking. PS: This is not a 100% alternative to constr but if all you want is regex validation, the above alternative works and makes mypy happy. :warning: pydantic v2 As mentioned in the comments, if using Pydantic V2, regex is replaced with pattern. install_component_version: str = Field(pattern=r"^[0-9]+.[0-9]+.[0-9]$")
Question: Pylint gives a warning whenever something like this happens: import typing SEQ_FR = typing.TypeVar("SEQ_FR") #^^^^^ gets underlined with the warning  The warning is like this: Type variable name "SEQ_FR" doesn't conform to predefined naming style. pylint(invalid-name) I tried searching through Pylint's documentations with no luck on finding the exact regex / specifications used. Doesn't seem like I can pass a custom regex onto Pylint for this as well, unlike regular variables, methods, functions, classes, etc. What is the specification used by Pylint to flag TypeVar variables as valid or invalid names?

Answer: You can find the rule used in the Pylint messages documentation; this error is named invalid-name, so the specific documentation can be found on the invalid-name / C0103 page, which has a TypeVar rule in the Predefined Naming Patterns section:  Name type: typevar Good Names: T, _CallableT, _T_co, AnyStr, DeviceTypeT, IPAddressT Bad Names: DICT_T, CALLABLE_T, ENUM_T, DeviceType, _StrType, TAnyStr  It doesn't document the exact regex rule here, but Pylint will actually include the regex used in the error message when you use the  --include-naming-hint=y command-line switch *):  Type variable name "SEQ_FR" doesn't conform to predefined naming style ('^_{0,2}(?!T[A-Z])(?:[A-Z]+|(?:[A-Z]+[a-z]+)+T?(?<!Type))(?:_co(?:ntra)?)?$' pattern) (invalid-name)  Alternatively, you can find the regex for typevars in the source code. Breaking the pattern down, typevar names are compliant when following these rules:  Optionally start with 0-2 underscores not starting with T<capital letter> either  all capital letters and no underscores, or a PascalCaseWord ✝) optionally ending in T, no underscores, and not ending with Type   with an optional _co or _contra ending.  Put differently, typevar names must be either PascalCase ✝) or all-caps, are optionally protected (_) or private (__), are optionally marked as covariant (_co) or contravariant (_contra), and should not end in Type. A compliant name for your example could be SeqFr or SeqFrT; the T suffix is meant to make it clear the PascalCaseT name is a typevar. Alternatively, you can specify your own regex with the  --typevar-rgx=<regex> command-line switch *). Note: As Pierre Sassoulas (the maintainer of Pylint) pointed out in a comment: There is no PEP-8 convention (yet) for typevar naming; instead the PyLint team captured the rules observed in Python projects and the type hinting documentation. The exact rule is therefore still subject to change if an official convention were to be created.  *) The Visual Studio Code settings for the Python extension include a python.linting.pylintArgs option that takes a list of command-line switches. ✝) PascalCase is also known as CapitalizedWords, UpperCamelCase or StudlyCase. Don't confuse this with camelCase (initial letter lowercase) or snake_case (all lowercase with underscores). I regularly do! When in doubt, Wikipedia has a handly table of multi-word formats.
Question: If you don't add any more fields to your subclass is there a need to add the @dataclass decorator to it and would it do anything? If there is no difference, which is the usual convention? from dataclasses import dataclass @dataclass class AAA: x: str y: str ... # decorate? class BBB(AAA): ... 

Answer: From documentation, dataclasses source code and experimenting with class instances, I don't see any difference, even if new fields are added to a subclass1. If we are talking conventions, then I would advise against decorating a subclass. Class BBB's definition is supposed to say "this class behaves just like AAA, but with these changes" (likely a couple of new methods in your case). Re-decorating BBB:  serves no purpose violates DRY principle: although it's just one line, but still there's no particular difference between re-decorating and copy-pasting a short method from superclass could (potentially) get in the way of changing AAA: you might theoretically switch to another library for dataclassing or decide to use @dataclass with non-default parameters, and that'd require maintaining subclass as well (for no good reason).   Update 1 As Andrius corrected me, if you add new fields in a subclass, you'll encounter the following problem: >>> class BBB(AAA): ... z: str ... >>> BBB('a', 'b', 'c') Traceback (most recent call last): File "<input>", line 1, in <module> BBB('a', 'b', 'c') TypeError: AAA.__init__() takes 3 positional arguments but 4 were given  Original question specifically says that new fields won't be added, so this is just a correction of my initial answer.
Question: The pyproject.toml specification affords the ability to specify the project version, e.g. [project] name = "foo" version = "0.0.1"  However, it is also a common Python idiom to put __version__ = "0.0.1" in foo/__init__.py so that users can query it. Is there a standard way of extracting the version from the pyproject.toml and getting it into the foo/__init__.py?

Answer: There are two approaches you can take here.  Keep version in pyproject.toml and get it from the package metadata in the source code. So, in your mypkg/__init__.py or wherever: from importlib.metadata import version __version__ = version("mypkg")  importlib.metadata.version is available since Python 3.8. For earlier Python versions, you can do similar with the importlib_metadata backport.  Keep the version in the source code and instruct the build system to get it from there. For a setuptools build backend, it looks like this in pyproject.toml: [project] name = "mypkg" dynamic = ["version"] [tool.setuptools.dynamic] version = {attr = "mypkg.__version__"}    My recommendation is (🥁) ... neither! Don't keep a __version__ attribute in the source code at all. It's an outdated habit which we can do without these days. Version is already a required field in the package metadata, it's redundant to keep the same string as an attribute in the package/module namespace.