eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Does the re module of Python3 offer an in-build way to get the match and the rest (none-match) back? Here is a simple example: >>> import re >>> p = r'\d' >>> s = '1a' >>> re.findall(p, s) ['1']  The result I want is something like ['1', 'a'] or [['1'], ['a']] or something else where I can differentiate between match and rest. Of course can subtract the resulting (matching) string from the original one to get the rest. But is there an in build way for this? I do not set the regex tag here because the question is less related to RegEx itself but more to a feature of a Python package.

Answer: Possible solution is the following: import re string = '1a' re_pattern = r'^(\d+)(.*)' result = re.findall(re_pattern, string) print(result)  Returns list of tuples [('1', 'a')]  or if you like to return list of str items result = [item for t in re.findall(re_pattern, string) for item in t] print(result)  Returns ['1', 'a']  Explanations to the code:  re_pattern = r'(\d+)(.*)' is looking for two groups: 1st group (\d+) means digits one or more, 2nd group (.*) means the rest of the string. re.findall(re_pattern, string) returns list of tuple like [('1', 'a')] list comprehension converts list of tuples to list of string items
Question: I want to rebuild the following logic with numpy broadcasting function such as np.where: From a 2d array check per row if the first element satisfies a condition. If the condition is true then return the first three elements as a row, else the last three elements. A short MWE in form of a for-loop which I want to circumvent: import numpy as np array = np.array([ [1, 2, 3, 4], [1, 2, 4, 2], [2, 3, 4, 6] ]) new_array = np.zeros((array.shape[0], array.shape[1]-1)) for i, row in enumerate(array): if row[0] == 1: new_array[i] = row[:3] else: new_array[i] = row[-3:] 

Answer: If you want to use np.where: import numpy as np array = np.array([ [1, 2, 3, 4], [1, 2, 4, 2], [2, 3, 4, 6] ]) cond = array[:, 0] == 1 np.where(cond[:, None], array[:,:3], array[:,-3:])  output: array([[1, 2, 3], [1, 2, 4], [3, 4, 6]])  EDIT slightly more concise version: np.where(array[:, [0]] == 1, array[:,:3], array[:,-3:])
Question: What is the loss function used in Trainer from the Transformers library of Hugging Face? I am trying to fine tune a BERT model using the Trainer class from the Transformers library of Hugging Face. In their documentation, they mention that one can specify a customized loss function by overriding the compute_loss method in the class. However, if I do not do the method override and use the Trainer to fine tine a BERT model directly for sentiment classification, what is the default loss function being use? Is it the categorical crossentropy? Thanks!

Answer: It depends! Especially given your relatively vague setup description, it is not clear what loss will be used. But to start from the beginning, let's first check how the default compute_loss() function in the Trainer class looks like. You can find the corresponding function here, if you want to have a look for yourself (current version at time of writing is 4.17). The actual loss that will be returned with default parameters is taken from the model's output values:   loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]  which means that the model itself is (by default) responsible for computing some sort of loss and returning it in outputs. Following this, we can then look into the actual model definitions for BERT (source: here, and in particular check out the model that will be used in your Sentiment Analysis task (I assume a BertForSequenceClassification model. The code relevant for defining a loss function looks like this: if labels is not None: if self.config.problem_type is None: if self.num_labels == 1: self.config.problem_type = "regression" elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int): self.config.problem_type = "single_label_classification" else: self.config.problem_type = "multi_label_classification" if self.config.problem_type == "regression": loss_fct = MSELoss() if self.num_labels == 1: loss = loss_fct(logits.squeeze(), labels.squeeze()) else: loss = loss_fct(logits, labels) elif self.config.problem_type == "single_label_classification": loss_fct = CrossEntropyLoss() loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) elif self.config.problem_type == "multi_label_classification": loss_fct = BCEWithLogitsLoss() loss = loss_fct(logits, labels)  Based on this information, you should be able to either set the correct loss function yourself (by changing model.config.problem_type accordingly), or otherwise at least be able to determine whichever loss will be chosen, based on the hyperparameters of your task (number of labels, label scores, etc.)
Question: I have a dataframe where one of the columns which is in string format looks like this  filename 0 Machine02-2022-01-28_00-21-45.blf.424 1 Machine02-2022-01-28_00-21-45.blf.425 2 Machine02-2022-01-28_00-21-45.blf.426 3 Machine02-2022-01-28_00-21-45.blf.427 4 Machine02-2022-01-28_00-21-45.blf.428  I want my column to look like this  filename 0 2022-01-28 00-21-45 424 1 2022-01-28 00-21-45 425 2 2022-01-28 00-21-45 426 3 2022-01-28 00-21-45 427 4 2022-01-28 00-21-45 428  I tried this code df['filename'] = df['filename'].str.extract(r"(\d{4}-\d{1,2}-\d{1,2})_(\d{2}-\d{2}-\d{2}).*\.(\d+)", r"\1 \2 \3")  I am getting this error, unsupported operand type(s) for &: 'str' and 'int'. Can anyone please tell me where I am doing wrong ?

Answer: please try this: df['filename'] = df['filename'].str.split('-',1).apply(lambda x:' '.join(x[1].split('_')).replace('.blf.',' '))
Question: Imagine we have an original API that returns a generator (it really is a mechanisms that brings pages/chunks of results from a server while the providing a simple generator to the user, and lets him iterate over these results one by one. For simplicity: # Original sync generator def get_results(): # fetch from server yield 1 yield 2 # fetch next page yield 3 yield 4 # ....  Now there is a need to implement an asyncio version of the API, however we need to keep the old API operational as well. This is where things get complicated, we kind of want to translate an async generator into sync one, but I can't find an elegant way to do that. The best I could make work so far is "fetch all result into a list first, then provide a fake sync generator on that list". Which kind of defeats the purpose: # Async generator async def get_results_async(): # await fetch from server yield 1 yield 2 # await fetch next page yield 3 yield 4 # .... # Backward compatible sync generator def get_results(): async def gather_all_results(): res = [] async for i in get_results_async(): res.append(i) return res res = asyncio.run(gather_all_results()) for i in res: yield i  Is there a better, more elegant way to do that without fetching all the results before returning them? Thanks

Answer: For the reason that asyncio is contagious, it's hard to write elegant code to integrate asyncio code into the old codes. For the scenario above, the flowing code is a little better, but I don't think it's elegant enough. async def get_results_async(): # await fetch from server yield 1 yield 2 # await fetch next page yield 3 yield 4 # .... # Backward compatible sync generator def get_results(): gen = get_results_async() while True: try: yield asyncio.run(gen.__anext__()) except StopAsyncIteration: break  And you can re-use your event loop and not to create a new one. async def get_results_async(): # await fetch from server yield 1 yield 2 # await fetch next page yield 3 yield 4 # .... # loop that you save in somewhere. loop = asyncio.get_event_loop() # Backward compatible sync generator def get_results(): gen = get_results_async() while True: try: yield loop.run_until_complete(gen.__anext__()) except StopAsyncIteration: break
Question: Has anyone gotten this error when importing psycopg2 after successful installation? ImportError: dlopen(/Users/chrishicks/Desktop/test/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 0x0002): tried: '/Users/chrishicks/Desktop/test/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/_psycopg.cpython-39-darwin.so' (no such file), '/usr/lib/_psycopg.cpython-39-darwin.so' (no such file)  I have tried installing psycopg2 and psycopg2-binary and have tried both while running iTerm in Rosetta.

Answer: Using this line should fix it: pip3.9 install psycopg2-binary --force-reinstall --no-cache-dir
Question: I am trying to use Structural Pattern Matching (PEP634) from Python 3.10, but Jedi language server doesn't support the syntax. I've heard Pylance is better, but I can't find any way to switch VS Code to Pylance. I've downloaded the default Python extension, but only the Jedi language server is running. How can I make the switch?  EDIT Adding a picture of trying to search for "pylance" so there's no confusion, it's not there at all. It seems to claim it's part of the Python extension, but the language server being used is always Jedi.  Python extension packs: 

Answer: I was using the open source version of VS Code which doesn't have all extensions. Switching to the proprietary version (available on the AUR) fixed my issue.
Question: My type checker moans at me when I use snippets like this one from the Pydantic docs: from datetime import datetime from pydantic import BaseModel, validator class DemoModel(BaseModel): ts: datetime = None # Expression of type "None" cannot be # assigned to declared type "datetime" @validator('ts', pre=True, always=True) def set_ts_now(cls, v): return v or datetime.now()  My workarounds so far have been: ts: datetime = datetime(1970, 1, 1) # yuck  ts: datetime = None # type: ignore  ts: Optional[datetime] = None # Not really true. `ts` is not optional.  Is there a preferred way out of this conundrum? Or is there a type checker I could use which doesn't mind this?

Answer: New answer Use a Field with a default_factory for your dynamic default value: from datetime import datetime from pydantic import BaseModel, Field class DemoModel(BaseModel): ts: datetime = Field(default_factory=datetime.now)  Your type hints are correct, the linter is happy and DemoModel().ts is not None. From the Field docs:  default_factory: a zero-argument callable that will be called when a default value is needed for this field. Among other purposes, this can be used to set dynamic default values.
Question: I have 2 Pydantic models (var1 and var2). The input of the PostExample method can receive data either for the first model or the second. The use of Union helps in solving this issue, but during validation it throws errors for both the first and the second model. How to make it so that in case of an error in filling in the fields, validator errors are returned only for a certain model, and not for both at once? (if it helps, the models can be distinguished by the length of the field A). main.py @app.post("/PostExample") def postExample(request: Union[schemas.var1, schemas.var2]): result = post_registration_request.requsest_response() return result  schemas.py class var1(BaseModel): A: str B: int C: str D: str class var2(BaseModel): A: str E: int F: str 

Answer: You could use Discriminated Unions (credits to @larsks for mentioning that in the comments). Setting a discriminated union, "validation is faster since it is only attempted against one model", as well as "only one explicit error is raised in case of failure". Working example is given below. Another approach would be to attempt parsing the models (based on a discriminator you pass as query/path param), as described in this answer (Option 1). Working Example app.py import schemas from fastapi import FastAPI, Body from typing import Union app = FastAPI() @app.post("/") def submit(item: Union[schemas.Model1, schemas.Model2] = Body(..., discriminator='model_type')): return item  schemas.py from typing import Literal from pydantic import BaseModel class Model1(BaseModel): model_type: Literal['m1'] A: str B: int C: str D: str class Model2(BaseModel): model_type: Literal['m2'] A: str E: int F: str  Test inputs - outputs #1 Successful Response #2 Validation error #3 Validation error # Request body # Request body # Request body { { { "model_type": "m1", "model_type": "m1", "model_type": "m2", "A": "string", "A": "string", "A": "string", "B": 0, "C": "string", "C": "string", "C": "string", "D": "string" "D": "string" "D": "string" } } } # Server response # Server response # Server response 200 { { "detail": [ "detail": [ { { "loc": [ "loc": [ "body", "body", "Model1", "Model2", "B" "E" ], ], "msg": "field required", "msg": "field required", "type": "value_error.missing" "type": "value_error.missing" } }, ] { } "loc": [ "body", "Model2", "F" ], "msg": "field required", "type": "value_error.missing" } ] }
Question: I have an app using Uvicorn with FastAPI. I have also some connections open (e.g. to MongoDB). I want to gracefully close these connections once some signal occurs (SIGINT, SIGTERM and SIGKILL). My server.py file: import uvicorn import fastapi import signal import asyncio from source.gql import gql app = fastapi.FastAPI() app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]) app.mount("/graphql", gql) # handle signals HANDLED_SIGNALS = ( signal.SIGINT, signal.SIGTERM ) loop = asyncio.get_event_loop() for sig in HANDLED_SIGNALS: loop.add_signal_handler(sig, _some_callback_func) if __name__ == "__main__": uvicorn.run(app, port=6900)  Unfortunately, the way I try to achieve this is not working. When I try to Ctrl+C in terminal, nothing happens. I believe it is caused because Uvicorn is started in different thread... What is the correct way of doing this? I have noticed uvicorn.Server.install_signal_handlers() function, but wasn't lucky in using it...

Answer: FastAPI allows defining event handlers (functions) that need to be executed before the application starts up, or when the application is shutting down. Thus, you could use the shutdown event, as described here: @app.on_event("shutdown") def shutdown_event(): # close connections here  Update Since startup and shutdown events are now deprecated (and might be removed in the future), one could use a lifespan function instead. Examples and details can be found in this answer, as well as here, here and here.
Question: I want to achieve a similar behavior as the library Dask does, it is possible to use pip to install dask, dask[dataframe], dask[array] and others. They do it by using the setup.py with a packages key like this. If I install only dask the dask[dataframe] is not installed and they warn you about this when executing the module. I found this in the poetry documentation but when I execute poetry build I only get one .whl file with all of the packages within. How can I package my module to be able to install specific parts of a library using poetry?

Answer: Actually the Dask example does not install subpackages separately, it just installs the custom dependencies separately as explained in this link. In order to accomplish the same behavior using poetry you need to use this (as mentioned by user @sinoroc in this comment) The example pyproject.toml from the poetry extras page is this: [tool.poetry] name = "awesome" [tool.poetry.dependencies] # These packages are mandatory and form the core of this package’s distribution. mandatory = "^1.0" # A list of all of the optional dependencies, some of which are included in the # below `extras`. They can be opted into by apps. psycopg2 = { version = "^2.7", optional = true } mysqlclient = { version = "^1.3", optional = true } [tool.poetry.extras] mysql = ["mysqlclient"] pgsql = ["psycopg2"]  By using poetry build --format wheel a single wheel file would be created. In order to install a specific set of extra dependencies using pip and the wheel file you should use: pip install "wheel_filename.whl[mysql]"
Question: In order to create an abstract property in Python one can use the following code: from abc import ABC, abstractmethod class AbstractClassName(ABC): @cached_property @abstractmethod def property_name(self) -> str: pass class ClassName(AbstractClassName): @property def property_name(self) -> str: return 'XYZ' >>> o = AbstractClassName() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Can't instantiate abstract class AbstractClassName with abstract method property_name >>> o = ClassName() >>> o.property_name 'XYZ'  which is what I have expected. I wanted to create an abstract cached property, so I tried the following: from abc import ABC, abstractmethod from functools import cached_property class AbstractClassName(ABC): @cached_property @abstractmethod def property_name(self) -> str: pass class ClassName(AbstractClassName): @cached_property def property_name(self) -> str: return 'XYZ'  However, this is not working as I expected: >>> o = AbstractClassName() >>> o.property_name >>> o = ClassName() >>> o.property_name 'XYZ'  Notice that this time it is allowing me to create an instance of an abstract class AbstractClassName. I am using Python 3.10. Is there any way to defined an abstract cached property?

Answer: Here is a possible solution from abc import ABC, abstractmethod from functools import cached_property class AbstractClassName(ABC): @cached_property def property_name(self) -> str: return self._property_name() @abstractmethod def _property_name(self) -> str: ... class ClassName(AbstractClassName): def _property_name(self) -> str: print("Heavy calculation") return "XYZ"  Test AbstractClassName() # raise a = ClassName() print(a.property_name) print(a.property_name) # Heavy calculation # XYZ # XYZ
Question: I'm attempting to replace an image in a Word 2019 .docx file using the following code in Python: from docxtpl import DocxTemplate tpl = DocxTemplate("C:\\temp\\replace_picture_tpl.docx") context = {} tpl.replace_pic('Sample.png','C:\\temp\\NewImage.png') tpl.render(context) tpl.save("C:\\temp\\TestOutput.docx")  I get the error ValueError: Picture Sample.png not found in the docx template. I created the document by inserting a random .png file into a blank Word document and saving it. I have double checked and confirmed that it is a .docx file. I then used the code above to attempt to change the picture, which didn't work. I looked around online for a couple of days and haven't been able to find anything to help yet. I've gone through the docxtpl templates and sample code and it seems like the code references a "descr" tag to locate 'Sample.png'. However when I view the replace_picture_tpl.docx XML file, the image doesn't have this tag. Any suggestions?

Answer: So, this worked for me using docxtpl and a template I modified in MS Word: Right click the image in MS Word, Select "View Alt Text":  Write "replace_me" as the Alt Text. Save and close. Then: from docxtpl import DocxTemplate tpl = DocxTemplate("sometemplate.docx") tpl.replace_pic("replace_me", "yourimage.png")  Definitely worked on MS Word for Mac 2022, Version 16.69 (23010700).
Question: I have read FastAPI's documentation about middlewares (specifically, the middleware tutorial, the CORS middleware section and the advanced middleware guide), but couldn't find a concrete example of how to write a middleware class which you can add using the add_middleware function (in contrast to a basic middleware function added using a decorator) there nor on this site. The reason I prefer to use add_middleware over the app based decorator, is that I want to write a middleware in a shared library that will be used by several different projects, and therefore I can't tie it to a specific FastAPI instance. So my question is: how do you do it?

Answer: As FastAPI is actually Starlette underneath, you could use BaseHTTPMiddleware that allows you to implement a middleware class (you may want to have a look at this post as well). Below are given two variants of the same approach on how to do that, where the add_middleware() function is used to add the middleware class. Please note that is currently not possible to use BackgroundTasks (if that's a requirement for your task) with BaseHTTPMiddleware—check #1438 and #1640 for more details. Alternatives can be found in this answer and this answer. Option 1 middleware.py from fastapi import Request class MyMiddleware: def __init__( self, some_attribute: str, ): self.some_attribute = some_attribute async def __call__(self, request: Request, call_next): # do something with the request object content_type = request.headers.get('Content-Type') print(content_type) # process the request and get the response response = await call_next(request) return response  app.py from fastapi import FastAPI from middleware import MyMiddleware from starlette.middleware.base import BaseHTTPMiddleware app = FastAPI() my_middleware = MyMiddleware(some_attribute="some_attribute_here_if_needed") app.add_middleware(BaseHTTPMiddleware, dispatch=my_middleware)  Option 2 middleware.py from fastapi import Request from starlette.middleware.base import BaseHTTPMiddleware class MyMiddleware(BaseHTTPMiddleware): def __init__( self, app, some_attribute: str, ): super().__init__(app) self.some_attribute = some_attribute async def dispatch(self, request: Request, call_next): # do something with the request object, for example content_type = request.headers.get('Content-Type') print(content_type) # process the request and get the response response = await call_next(request) return response  app.py from fastapi import FastAPI from middleware import MyMiddleware app = FastAPI() app.add_middleware(MyMiddleware, some_attribute="some_attribute_here_if_needed")
Question: The discord bot I made a year ago and deployed to Heroku has worked until now. However, after changing some cogs and updating python to version 3.9.10, I get the following warning in the Heroku logs: app[worker.1]: /app/m_bot.py:120: RuntimeWarning: coroutine 'BotBase.load_extension' was never awaited app[worker.1]: client.load_extension(f"cogs.{filename[:-3]}") app[worker.1]: RuntimeWarning: Enable tracemalloc to get the object allocation traceback app[worker.1]: Bot is ready. app[api]: Build succeeded>  The 120 line block is: for filename in os.listdir("./cogs"): if filename.endswith(".py"): # cut of the .py from the file name client.load_extension(f"cogs.{filename[:-3]}")  The bot goes online but doesn't respond to any command. I haven't made any other changes apart from what was listed above. It works when I run my bot on my PC, so I suspect it might be a version problem. How can I resolve this?

Answer: Explanation As of discord.py version 2.0, Bot.load_extension is now a coroutine and has to be awaited. This is to allow Cog subclasses to override cog_unload with a coroutine. Code await must be used in front of client.load_extension, as shown: await client.load_extension("your_extension")  In each of your cogs: Replace the standard setup function with an asynchronous one: async def setup(bot): await bot.add_cog(YourCog(bot))   If you want to use the normal convention for adding extensions, you'll need to use the following code: In your client's file: async def load_extensions(): for filename in os.listdir("./cogs"): if filename.endswith(".py"): # cut off the .py from the file name await client.load_extension(f"cogs.{filename[:-3]}")  You should also wrap your login in an asynchronous 'main' function, where you would call this function. Note that the code below does not setup logging, you need to do so yourself: async def main(): async with client: await load_extensions() await client.start('your_token') asyncio.run(main())  These two functions replace the old way: client.run("your_token")  along with the code you posted in your question. Reference discord.py 2.0 async changes (Thank you ChrisDewa for mentioning this in your comment)