eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a function one of whose arguments is expected to be a type hint: something like typing.List, or typing.List[int], or even just int. Anything you would reasonably expect to see as a type annotation to an ordinary field. What's the correct type hint to put on this argument? (The context for this is that I'm writing a utility that works on classes that define fields using type annotations, a bit like the dataclass decorator.)

Answer: Answer for Python 3.10: Almost complete but less readable answer: type | types.GenericAlias | types.UnionType | typing._BaseGenericAlias | typing._SpecialForm  Here are the possibilities of all types of annotations I can think of:  Type object itself, such as int, list, etc. Corresponds to type. Type hinting generics in standard collections, such as list[int]. Corresponds to types.GenericAlias. Types union, such as int | float (but typing.Union[int, float] is not, it corresponds to the next item). Corresponds to types.UnionType. Generic concrete collections in typing, such as typing.List, typing.List[int], etc. Here I choose the common base class typing._BaseGenericAlias that first appeared in their inheritance chain. (In fact, not only these, but almost all subscriptible annotation classes in typing inherit from this class, including typing.Literal[True], typing.Union[int, float], etc.) Special typing primitives in typing, such as typing.Any, typing.NoReturn, etc. Corresponds to typing._SpecialForm.  It should be noted that the last two types begin with a single underline, indicating that they are internal types, and should not be imported and used from typing. However, they should be indispensable if you insist on completely covering all type annotations.  Python 3.11 update: I noticed that typing.Any has changed from typing._SpecialForm to typing._AnyMeta (to allow inheritance of it), but it is now a type (which we have already included) and we don't need to add anything new to it. typing.Self and typing.LiteralString have been added in 3.11, but they are both typing._SpecialForm, so there is also no need for anything new.  Python 3.12 update: We now have a type statement, which allows us to create typing.TypeAliasType object, this seems to be the only new content.
Question: I would like to turn the loop over the items in L in following code: L = [1,2,5,2,1,1,3,4] L_unique = [] for item in L: if item not in L_unique: L_unique.append(item)  to a list comprehension like: L_unique = [ item for item in L if item not in ???self??? ]  Is this possible in Python? And if it is possible, how can it be done?

Answer: It's possible. Here's a hack that does it, but I wouldn't use this in practice as it's nasty and depends on implementation details that might change and I believe it's not thread-safe, either. Just to demonstrate that it's possible. You're mostly right with your "somewhere must exist an object storing the current state of the comprehension" (although it doesn't necessarily have to be a Python list object, Python could store the elements some other way and create the list object only afterwards). We can find the new list object in the objects tracked by garbage collection. Collect the IDs of lists before the comprehension's list is created, then look again and take the one that wasn't there before. Demo: import gc L = [1,2,5,2,1,1,3,4] L_unique = [ item # the hack to get self for ids in ({id(o) for o in gc.get_objects() if type(o) is list},) for self in (o for o in gc.get_objects() if type(o) is list and id(o) not in ids) for item in L if item not in self ] print(L_unique)  Output (Attempt This Online!): [1, 2, 5, 3, 4]  Tested and worked in several versions from Python 3.7 to Python 3.11. For an alternative with the exact style you asked, only replacing your ???self???, see Mechanic Pig's updated answer.
Question: When viewing a Quilt data package in the catalog view, how do I ensure that the relevant information and data to my users bubbles up from N-depth of folders/files to the data package landing view?

Answer: [Disclaimer: I originally wrote this answer when I was working at Quilt Data] Create a file called quilt_summarize.json [Reference] which is a configuration file that renders one or more data-package elements in both Bucket view and Packages view. The contents of quilt_summarize.json are a JSON array of files that you wish to preview in the catalog, from any depth in your data package. Each file may be represented as a string or, if you wish to provide more configuration, as an object. You can preview all sorts of different files out-of-the-box including JSON, CSV, TXT, MD, XLS, XLSX, PARQUET, TSV, visualization libraries (including Vega, Altair, eCharts, and Voila), and iPython notebooks. Here's the syntax: // quilt_summarize.json [ "file1.json", "root_folder/second_level/third_level/file2.csv", "notebooks/file3.ipynb" ]  You can also define multi-column outputs: [ "file1.json", [{ "path": "file2.csv", "width": "200px" }, { "path": "file3.ipynb", "title": "Scientific notebook", "description": "[See docs](https://docs.com)" }] ]  which will render the file file1.json first (and span the whole width of the screen), then the viewport will be split into N columns (based on N-length of the JSON array) in the widths defined in the attributes.
Question: How do you update multiple properties on a pydantic model that are validated together and dependent upon each other? Here is a contrived but simple example: from pydantic import BaseModel, root_validator class Example(BaseModel): a: int b: int @root_validator def test(cls, values): if values['a'] != values['b']: raise ValueError('a and b must be equal') return values class Config: validate_assignment = True example = Example(a=1, b=1) example.a = 2 # <-- error raised here because a is 2 and b is still 1 example.b = 2 # <-- don't get a chance to do this  Error: ValidationError: 1 validation error for Example __root__ a and b must be equal (type=value_error)  Both a and b having a value of 2 is valid, but they can't be updated one at a time without triggering the validation error. Is there a way to put the validation on hold until both are set? Or a way to somehow update both of them at the same time? Thanks!

Answer: I found a couple solutions that works well for my use case.  manually triggering the validation and then updating the __dict__ of the pydantic instance directly if it passes -- see update method a context manager that delays validation until after the context exits -- see delay_validation method  from pydantic import BaseModel, root_validator from contextlib import contextmanager import copy class Example(BaseModel): a: int b: int @root_validator def enforce_equal(cls, values): if values['a'] != values['b']: raise ValueError('a and b must be equal') return values class Config: validate_assignment = True def update(self, **kwargs): self.__class__.validate(self.__dict__ | kwargs) self.__dict__.update(kwargs) @contextmanager def delay_validation(self): original_dict = copy.deepcopy(self.__dict__) self.__config__.validate_assignment = False try: yield finally: self.__config__.validate_assignment = True try: self.__class__.validate(self.__dict__) except: self.__dict__.update(original_dict) raise example = Example(a=1, b=1) # ================== This didn't work: =================== # example.a = 2 # <-- error raised here because a is 2 and b is still 1 # example.b = 2 # <-- don't get a chance to do this # ==================== update method: ==================== # No error raised example.update(a=2, b=2) # Error raised as expected - a and b must be equal example.update(a=3, b=4) # Error raised as expected - a and b must be equal example.update(a=5) # # =============== delay validation method: =============== # No error raised with example.delay_validation(): example.a = 2 example.b = 2 # Error raised as expected - a and b must be equal with example.delay_validation(): example.a = 3 example.b = 4 # Error raised as expected - a and b must be equal with example.delay_validation(): example.a = 5
Question: I have a FastAPI app that gets an image from an API. This image is stored in a variable with type: bytes. I want to display the image in HTML/Jinja2 template (without having to download it). I followed many tutorials but couldn't find the solution. Here is what I came up with so far: @app.get("/{id}") async def root(request: Request, id: str): picture = await get_online_person() data = base64.b64encode(picture) # convert to base64 as bytes data = data.decode() # convert bytes to string # str_equivalent_image = base64.b64encode(img_buffer.getvalue()).decode() img_tag = '<img src="data:image/png;base64,{}">'.format(data) return templates.TemplateResponse( "index.html", {"request": request, "img": img_tag} )  All I get in the HTML is this: (as text on the page, not from source code)  <img src="data:image/png;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQICAQECAQEBAgICAgICAgICAQICAgICAgICAgL/2wBDAQEBAQEBAQEBAQECAQEBAgICAgI CAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgL/wgARCAQABAADASIAA hEBAxEB/8QAHgAAAQUBAQEBAQAAAAAAAAAABQIDBAYHAQgACQr/xAAcAQACAwEBAQEAAAAAAAAAAAACAwABBAUGBwj/2gAMAwEAAhADEAAAAfEpwSR+a+9IPR3c7347iwscmWyYchEIJjn+MbJj/c4FFbbb9J5....................  Note: For people who are marking my question to a duplicate talking about urllib, I cannot use urllib because the image I'm getting is from ana API, and using their direct url will result in a 403 Forbidden, so I should use their python API to get the image.

Answer: On server side—as shown in the last section of this answer—you should return only the base64-encoded string in the context of the TemplateResponse (without using the <img> tag, as shown in your question): # ... base64_encoded_image = base64.b64encode(image_bytes).decode("utf-8") return templates.TemplateResponse("index.html", {"request": request, "myImage": base64_encoded_image})  On client side, you could display the image as follows: <img src="data:image/jpeg;base64,{{ myImage | safe }}">  Alternative approaches can be found here.
Question: In a few of my beginner projects this strange red line underscoring one or more of my imports keeps appearing almost randomly and I can't figure out why. As the module is working perfectly fine it shouldn't have something to do regarding which Folder I open VS Code in as it can get resolved, so sys.path should also have the right path, as far as I'm concerned. Sometimes it works when I switch my imports around but often it just underscores a single import or switching them around doesn't do anything. Also when I try to let VS Code sort them with isort, nothing happens and nothing had ever happened.

Answer: Edit: I realized that recently the isort extension from Microsoft was automatically added to my extensions and this has caused the annoying error to start showing. It may be that the extension is conflicting somehow with the isort library installed in your venv. The extension isn't needed, so I've just disabled it and no longer encounter this error. It seems like this error started happening after I recently updated my VS Code to 1.73.0 (Insiders). I was able to get around it by splitting up my imports so that they don't get auto-formatted to be on multiple lines. Here's an example: Before the "fix", notice the squiggly red line with the annoying error:  After the "fix", no more squiggly red line:
Question: I am trying to loop through all mp3 files in my directory in MacOS Monterrey and for every iteration get the file's more info attributes, like Title, Duration, Authors etc. I found a post saying use xattr, but when i create a variable with xattr it doesn't show any properties or attributes of the files. This is in Python 3.9 with xattr package import os import xattr directory = os.getcwd() for filename in os.listdir(directory): f = os.path.join(directory, filename) # checking if it is a file if os.path.isfile(f): print(f) x = xattr.xattr(f) xs = x.items() 

Answer: xattr is not reading mp3 metadata or tags, it is for reading metadata that is stored for the particular file to the filesystem itself, not the metadata/tags thats stored inside the file. In order to get the data you need, you need to read the mp3 file itself with some library that supports reading ID3 of the file, for example: eyed3. Here's a small example: from pathlib import Path import eyed3 root_directory = Path(".") for filename in root_directory.rglob("*.mp3"): mp3data = eyed3.load(filename) if mp3data.tag != None: print(mp3data.tag.artist) print(mp3data.info.time_secs)
Question: I'm working with a protobuf message that has some of the fields marked for deprecation with [deprecated = true]. To my understanding the field can still be used by some part of the code (maybe with a warning). I want to make sure that my code is still supporting this field with the possibility of handling the case when it actually gets deprecated. Was thinking HasField gives me that tool but sounds like HasField only check if an existing field in a message has been set or not. In my case my proto message looks roughly like this: message Message1 { map<string, Message2> message_collection = 1; } message Message2 { bool some_var = 1 [deprecated = true]; }  I was hoping for a piece of code like this: my_message = Message1() for mystr, mymessage2 in my_message.message_collection.items(): if mymessage2.HasField("some_var"): mymessage2.some_var = True  How can I check if some_var in Message2 is still a defined field or not?

Answer: Deprecated fields have no impact on code, as per the docs. In your case, it looks like you are trying to evaluate if the class Message2 has a field defined or not? Before going forward, you need add optional as an option in your proto to allow for HasField() to work: my_message.proto syntax = "proto3"; message Message1 { map<string, Message2> message_collection = 1; } message Message2 { optional bool some_var = 1 [deprecated = true]; }  Your python code should look like the following: main.py from sample_message_pb2 import Message1, Message2 # Create your object my_message = Message1( message_collection={ 'key_1': Message2(some_var=True), 'key_2': Message2(some_var=False), 'key_3': Message2() } ) # Iterate over your object for mystr, mymessage2 in my_message.message_collection.items(): # If the value is assigned or not if mymessage2.HasField(field_name): print(f'{mystr}: {field_name} = {getattr(mymessage2, field_name)}') # Reassign the value mymessage2.some_var = True # If the field exists or not field_names = mymessage2.ListFields() if 'pickles' in field_names: print('Uh oh, here comes Mr. Pickles')  I dug around and could not find if it's possible to surface the deprecation warning to your python code. Looks like it's a limitation presently.
Question: This is a follow up question for this answer. For a conda environment specification file environment.yml, if the variable that I am defining is PATH, for example, how can I prepend or append to it instead of just overwriting it? Is the following correct? name: foo channels: - defaults dependencies: - python variables: MY_VAR: something OTHER_VAR: ohhhhya PATH: /some/path:$PATH 

Answer: It dependes on whether you are using windows or linux. A look at the source code of the environment init code reveals that conda itself simply executes bash (linux) or cmd.exe (win) calls: linux: yield from (f"export {envvar}='{value}'" for envvar, value in sorted(env_vars.items()))  windows: yield from (f'@SET "{envvar}={value}"' for envvar, value in sorted(env_vars.items()))  So make sure that you are using the correct syntax for your variable. In case of linux, that would be variables: MY_VAR: something OTHER_VAR: ohhhhya PATH: /some/path $PATH  AFAIK windows uses ; to delimit entries, so you would probably have to do this (untested): variables: MY_VAR: something OTHER_VAR: ohhhhya PATH: /some/path;%PATH%
Question: I'm trying to solve a Sylvester matrix equation of the form  AX + XB = C  From what I've seen, these equations are usually solved with the Bartels-Stewart algorithm taking successive Schur decompositions. I'm aware scipy.linalg already has a solve_sylvester function, but I'm integrating the solution to the Sylvester equation into a neural network, so I need a way to calculate gradients to make A, B, and C learnable. Currently, I'm just solving a linear system with torch.linalg.solve using the Kronecker product and vectorization trick, but this has terrible runtime complexity. I haven't found any PyTorch support for Sylvester equations, let alone Schur decompositions, but before I try to implement Barters-Stewart on the GPU, is there a simpler way to find the gradients?

Answer: Initially I wrote a solution that would give complex X based on Bartels-Stewart algorithm for the m=n case. I had some problems because the eigenvector matrix is not accurate enough. Also the real part gives the real solution, and the imaginary part must be a solution for AX - XB = 0 import torch def sylvester(A, B, C, X=None): m = B.shape[-1]; n = A.shape[-1]; R, U = torch.linalg.eig(A) S, V = torch.linalg.eig(B) F = torch.linalg.solve(U, (C + 0j) @ V) W = R[..., :, None] - S[..., None, :] Y = F / W X = U[...,:n,:n] @ Y[...,:n,:m] @ torch.linalg.inv(V)[...,:m,:m] return X.real if all(torch.isreal(x.flatten()[0]) for x in [A, B, C]) else X  As can be verified on the GPU with device='cuda' # Try different dimensions for batch_size, M, N in [(1, 4, 4), (20, 16, 16), (6, 13, 17), (11, 29, 23)]: print(batch_size, (M, N)) A = torch.randn((batch_size, N, N), dtype=torch.float64, device=device, requires_grad=True) B = torch.randn((batch_size, M, M), dtype=torch.float64, device=device, requires_grad=True) X = torch.randn((batch_size, N, M), dtype=torch.float64, device=device, requires_grad=True) C = A @ X - X @ B X_ = sylvester(A, B, C) C_ = (A) @ X_ - X_ @ (B) print(torch.max(abs(C - C_))) X.sum().backward()  A faster algorithm, but inaccurate in the current pytorch version is def sylvester_of_the_future(A, B, C): def h(V): return V.transpose(-1,-2).conj() m = B.shape[-1]; n = A.shape[-1]; R, U = torch.linalg.eig(A) S, V = torch.linalg.eig(B) F = h(U) @ (C + 0j) @ V W = R[..., :, None] - S[..., None, :] Y = F / W X = U[...,:n,:n] @ Y[...,:n,:m] @ h(V)[...,:m,:m] return X.real if all(torch.isreal(x.flatten()[0]) for x in [A, B, C]) else X  I will leave it here maybe in the future it will work properly.
Question: I'm trying to do a very simple thing: to load a dataset from the Huggingface library (see example code here) on my Mac: from datasets import load_dataset raw_datasets = load_dataset("glue", "mrpc")  I'm getting the following error: PermissionError: [Errno 13] Permission denied: '/Users/username/.cache/huggingface/datasets/downloads/6d9bc094a0588d875caee4e51df39ab5d6b6316bf60695294827b02601d421a5.759f3e257a3fad0984d9f8ba9a26479d341795eb50fa64e4c1de40f1fc421313.py.lock'  I've just spent an hour googling solutions for this, but so far nothing has worked. Can anyone help? Thanks in advance!

Answer: OK, I managed to solve it by manually changing the permissions of the right folders on my Mac:  I navigated to the /Users/username/.cache/huggingface/datasets/downloads folder in the Finder (you can see hidden files and folders such as ".cache" by pressing " command shift + . ") Then I went to the info window for this 'downloads' folder (" command i "), clicked 'Sharing & Permissions', clicked the lock to make changes and then gave everyone read & write access (I'm the only one using this computer, so it doesn't matter) I then got some new permission errors relating to some other folders, so I just went to each folder and changed the permissions for all of them  Not sure why this worked and the 'chmod 777' command didn't, but I'm glad it did. Thank you @Cuartero for pointing me in the right direction!
Question: When using numpy I can use np.vectorize to vectorize a function that contains if statements in order for the function to accept array arguments. How can I do the same with torch in order for a function to accept tensor arguments? For example, the final print statement in the code below will fail. How can I make this work? import numpy as np import torch as tc def numpy_func(x): return x if x > 0. else 0. numpy_func = np.vectorize(numpy_func) print('numpy function (scalar):', numpy_func(-1.)) print('numpy function (array):', numpy_func(np.array([-1., 0., 1.]))) def torch_func(x): return x if x > 0. else 0. print('torch function (scalar):', torch_func(-1.)) print('torch function (tensor):', torch_func(tc.tensor([-1., 0., 1.]))) 

Answer: You can use .apply_() for CPU tensors. For CUDA ones, the task is problematic: if statements aren't easy to SIMDify. You may apply the same workaround for functorch.vmap as video drivers used to do for shaders: evaluate both branches of the condition and stick to arithmetics. Otherwise, just use a for loop: that's what np.vectorize() mostly does anyway. def torch_vectorize(f, inplace=False): def wrapper(tensor): out = tensor if inplace else tensor.clone() view = out.flatten() for i, x in enumerate(view): view[i] = f(x) return out return wrapper
Question: I'm trying to setup a way to debug a Streamlit script in PyCharm. I'm on a Win10/64bit machine, working within an virtual environment created with conda. Running the code in the default way with streamlit run main.py works as expected. I have already read several forum posts and most importantly this related question on SO here. My question is the following: The answer in the question above suggests to change the debug configuration to use Module name instead of Script path and enter streamlit.cli as module. Then in the parameters one should set run main.py as the argument. Unfortunately this gets me the following error:  No module named streamlit.cli  Where do I find streamlit.cli shouldn't it be installed along with the default pip install of the library? Do I need to install it separately? Any help is much appreciated!

Answer: Stumbled across the (rather simple) answer by accident: Simply use the "correct" module name, which in my case was streamlit instead of streamlit.cli. So for debugging I now have the following configuration:  Module Name (instead of "Script path"): streamlit Parameter: run main.py Interpreter options: not set Working directory: <path\to\main.py>  As I have read in some streamlit forum post: "Works like a (Py)Charm!" :)
Question: These are the demo models class Author(models.Model): name = models.CharField(max_lenght=5) class Post(models.Model): author = models.ForeignKey(Author, on_delete=models.CASCADE) title = models.CharField(max_lenght=50) body = models.TextField()  And the respective views are class AuthorViewSet(viewsets.ModelViewSet): queryset = Author.objects.all() serializer_class = AuthorSerializer class PostViewSet(viewsets.ModelViewSet): queryset = Post.objects.all() serializer_class = PostStatSerializer  I am trying to perform an update/put action on PostViewSet and which is succesfull, but I am expecting different output. After successful update of Post record, I want to send its Author record as output with AuthorSerializer. How to override this and add this functionality?

Answer: I figured out some less code fix for my issue. class PostViewSet(viewsets.ModelViewSet): queryset = Post.objects.all() serializer_class = PostStatSerializer def update(self, request, *args, **kwargs): super().update(request, *args, **kwargs) instance = self.get_object() return Response(AuthorSerializer(instance.author).data)