eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I recently read a paper entitled "REGULARIZING NEURAL NETWORKS BY PENALIZING CONFIDENT OUTPUT DISTRIBUTIONS https://arxiv.org/abs/1701.06548". The authors discuss regularizing neural networks by penalizing low entropy output distributions through adding a negative entropy term to the negative log-likelihood and creating a custom loss function for model training.  The value β controls the strength of confidence penalty. I have written a custom function for categorical cross-entropy as shown below but the negative entropy term need to be added to the loss function. import tensorflow as tf def custom_loss(y_true, y_pred): cce = tf.keras.losses.CategoricalCrossentropy() cce_loss = cce(y_true, y_pred) return cce_loss 

Answer: The entropy of y_pred is essentially the categorical cross entropy between y_pred and itself:  def custom_loss(y_true, y_pred, beta): cce = tf.keras.losses.CategoricalCrossentropy() return cce(y_true, y_pred) - beta*cce(y_pred, y_pred)
Question: I'm trying to take my df.to_csv which is sep="\t" and turn that tab into two spaces instead. This question is similar but the solution isn't working: Pandas to_csv with multiple separators \s+ won't work as python will complain that its not a single char separator. This works as its a tab: df2.to_csv('test.csv', index=False, sep='\t', quoting=csv.QUOTE_NONE, quotechar="", escapechar=None)  this throws TypeError: "delimiter" must be a 1-character string df2.to_csv('test.csv', index=False, sep='\s+', quoting=csv.QUOTE_NONE, quotechar="", escapechar=None) 

Answer: Let's look at using to_markdown instead of to_csv: df = pd.DataFrame({'col1':'aaa bbb ccc'.split(), 'col2':[1, 10, 1000], 'col3': [True, False, True]}) df.to_markdown('a.txt', tablefmt='plain', index=False) !type a.txt  File: col1 col2 col3 aaa 1 True bbb 10 False ccc 1000 True
Question: I am using FastAPI and I have a number of endpoints that look like this: @app.get("/REDS/") def query_REDS( request: Request, lighter: Optional[bool] = False, darker: Optional[bool] = False, inverse: Optional[bool] = False, amount: Optional[int] = 10): pass # Work done here @app.get("/BLUES/") def query_BLUES( request: Request, lighter: Optional[bool] = False, darker: Optional[bool] = False, inverse: Optional[bool] = False, amount: Optional[int] = 10): pass # Work done here @app.get("/GREENS/") def query_GREENS( request: Request, lighter: Optional[bool] = False, darker: Optional[bool] = False, inverse: Optional[bool] = False, amount: Optional[int] = 10): pass # Work done here  This looks like this in the swagger UI:  The real config is passed in the request and parsed manually. Whenever I need to update the signature of these endpoints, I need to update it in like 20 different places. Is there a way to define those specific default arguments in one place? I tried using the pydantic BaseModel to define an input model: class Arguments(BaseModel): lighter: Optional[bool] = False darker: Optional[bool] = False inverse: Optional[bool] = False amount: Optional[int] = 10 @app.get("/REDS/") def query_REDS( request: Request, arguments: Arguments): pass # Work done here @app.get("/BLUES/") def query_BLUES( request: Request, arguments: Arguments): pass # Work done here @app.get("/GREENS/") def query_GREENS( request: Request, arguments: Arguments): pass # Work done here.  But this is not what I am after, first of all because using a body in a get request is not recommended and not supported everywhere and second of all because it is not that useful in the swagger UI:  Is there a way to define a sort of default signature to a number of different enpoints?

Answer: To do what you want, you can use regular classes or pydantic models as class dependencies: class CommonParams: def __init__(self, request: Request, lighter: Optional[bool] = False, darker: Optional[bool] = False, inverse: Optional[bool] = False, amount: Optional[int] = 10): self.request = request self.lighter = lighter self.darker = darker self.inverse = inverse self.amount = amount class Arguments(BaseModel): lighter: Optional[bool] = False darker: Optional[bool] = False inverse: Optional[bool] = False amount: Optional[int] = 10 @app.get("/REDS/") def query_REDS(params=Depends(CommonParams)): pass # Work done here @app.get("/BLUES/") def query_BLUES(params=Depends(Arguments)): pass # Work done here
Question: I'm running into this static type hint mismatch (with Pyright): from __future__ import annotations from typing import AnyStr, Iterable def foo(i: Iterable[AnyStr]): return i def bar(i: Iterable[str] | Iterable[bytes]): return i def baz(i: Iterable[str | bytes]): return i def main(): s = ['a'] # makes sense to me baz(foo(s)) # allowed foo(baz(s)) # not allowed # makes sense to me baz(bar(s)) # allowed bar(baz(s)) # not allowed bar(foo(s)) # allowed foo(bar(s)) # nope -- why?  What's the difference between Iterable[AnyStr] and Iterable[str] | Iterable[bytes]? Shouldn't they be "equivalent"? (save for AnyStr referring to a single consistent type within a context) More concretely: what is the right way to type-hint the following? import random from typing import Iterable, AnyStr def foo(i: Iterable[AnyStr]): return i def exclusive_bytes_or_str(): # type-inferred to be Iterator[bytes] | Iterator[str] if random.randrange(2) == 0: return iter([b'bytes']) else: return iter(['str']) foo(iter([b'bytes'])) # fine foo(iter(['str'])) # fine foo(exclusive_bytes_or_str()) # same error 

Answer: Paraphrased answer from erictraut@github:  This isn't really the intended use for a constrained TypeVar. I recommend using an @overload instead: @overload def foo(i: Iterable[str]) -> Iterable[str]: ... @overload def foo(i: Iterable[bytes]) -> Iterable[bytes]: ... def foo(i: Iterable[AnyStr]) -> Iterable[AnyStr]: return i   Because:  The type Iterable[str] | Iterable[bytes] is not assignable to type Iterable[AnyStr]. A constrained type variable needs to be matched against one of its contraints, not multiple constraints. When a type variable is "solved", it needs to be replaced by another (typically concrete) type. If foo(bar(s)) were allowed, what type would the AnyType@foo type variable resolve to? If it were resolved to type str | bytes, then the concrete return type of foo would be Iterable[str | bytes]. That's clearly wrong.
Question: Tried plotting a pie chart using: import pandas as pd import numpy as np data = {'City': ['KUMASI', 'ACCRA', 'ACCRA', 'ACCRA', 'KUMASI', 'ACCRA', 'ACCRA', 'ACCRA', 'ACCRA'], 'Building': ['Commercial', 'Commercial', 'Industrial', 'Commercial', 'Industrial', 'Commercial', 'Commercial', 'Commercial', 'Commercial'], 'LPL': ['NC', 'C', 'C', 'C', 'NC', 'C', 'NC', 'NC', 'NC'], 'Lgfd': ['NC', 'C', 'C', 'C', 'NC', 'C', 'NC', 'NC', 'C'], 'Location': ['NC', 'C', 'C', 'C', 'NC', 'C', 'C', 'NC', 'NC'], 'Hazard': ['NC', 'C', 'C', 'C', 'NC', 'C', 'C', 'NC', 'NC'], 'Inspection': ['NC', np.nan, np.nan, np.nan, 'NC', 'NC', 'C', 'C', 'C'], 'Name': ['Zonal', 'In Prog', 'Tullow Oil', 'XGI', 'Food Factory', 'MOH', 'EV', 'CSD', 'Electroland'], 'Air Termination System': ['Vertical Air Termination', 'Vertical Air Termination', 'Vertical Air Termination', 'Early Streamer Emission', 'Vertical Air Termination', 'Vertical Air Termination', 'Vertical Air Termination', 'Vertical Air Termination', 'Early Streamer Emission'], 'Positioned Using': ['Highest Points', 'Software', 'Software', 'Software', 'Highest Points', np.nan, np.nan, 'Rolling Sphere Method', 'Software']} df = pd.DataFrame(data) colors = ['#ff9999','#66b3ff','#99ff99','#ffcc99'] data = df["Air Termination System"].value_counts().plot(kind="pie",autopct='%1.1f%%', radius=1.5, shadow=True, explode=[0.05, 0.05], colors=colors)  Present chart looks like:  How do I bring the title "Air Termination Systems" outside the chart and also create a legend at the top right using the colors?

Answer:  legend=True adds the legend title='Air Termination System' puts a title at the top ylabel='' removes 'Air Termination System' from inside the plot. The label inside the plot was a result of radius=1.5 labeldistance=None removes the other labels since there is a legend. If necessary, specify figsize=(width, height) inside data.plot(...)  colors = ['#ff9999','#66b3ff','#99ff99','#ffcc99'] data = df["Air Termination System"].value_counts() ax = data.plot(kind="pie", autopct='%1.1f%%', shadow=True, explode=[0.05, 0.05], colors=colors, legend=True, title='Air Termination System', ylabel='', labeldistance=None) ax.legend(bbox_to_anchor=(1, 1.02), loc='upper left') plt.show()
Question: Consider the following code import contextlib import abc import asyncio from typing import AsyncContextManager, AsyncGenerator, AsyncIterator class Base: @abc.abstractmethod async def subscribe(self) -> AsyncContextManager[AsyncGenerator[int, None]]: pass class Impl1(Base): @contextlib.asynccontextmanager async def subscribe(self) -> AsyncIterator[ AsyncGenerator[int, None] ]: <-- mypy error here async def _generator(): for i in range(5): await asyncio.sleep(1) yield i yield _generator()  For Impl1.subscribe mypy gives the error Signature of "subscribe" incompatible with supertype "Base"  What is the correct way to specify type hints in the above case? Or is mypy wrong here?

Answer: I just happened to come up with the same problem and found this question on the very same day, but also figured out the answer quickly. You need to remove async from the abstract method. To explain why, I'll simplify the case to a simple async iterator: @abc.abstractmethod async def foo(self) -> AsyncIterator[int]: pass async def v1(self) -> AsyncIterator[int]: yield 0 async def v2(self) -> AsyncIterator[int]: return v1()  If you compare v1 and v2, you'll see that the function signature looks the same, but they actually do very different things. v2 is compatible with the abstract method, v1 is not. When you add the async keyword, mypy infers the return type of the function to be a Coroutine. But, if you also put a yield in, it then infers the return type to be AsyncIterator: reveal_type(foo) # -> typing.Coroutine[Any, Any, typing.AsyncIterator[builtins.int]] reveal_type(v1) # -> typing.AsyncIterator[builtins.int] reveal_type(v2) # -> typing.Coroutine[Any, Any, typing.AsyncIterator[builtins.int]]  As you can see, the lack of a yield in the abstract method means that this is inferred as a Coroutine[..., AsyncIterator[int]]. In other words, a function used like async for i in await v2():. By removing the async: @abc.abstractmethod def foo(self) -> AsyncIterator[int]: pass reveal_type(foo) # -> typing.AsyncIterator[builtins.int]  We see that the return type is now AsyncIterator and is now compatible with v1, rather than v2. In other words, a function used like async for i in v1(): You can also see that this is fundamentally the same thing as v1: def v3(self) -> AsyncIterator[int]: return v1()  While the syntax is different, both v3 and v1 are functions which will return an AsyncIterator when called, which should be obvious given that we are literally returning the result of v1().
Question: I am quite new to using Pydantic. The Issue I am facing right now is that the Model Below is not raising the Expected Exception when the value is out of range. For example, if you pass -1 into this model it should ideally raise an HTTPException. but nothing happens I am not sure where I might be going wrong. Any Advice would be great. class GetInput: """ for the fields endpoint """ def __init__(self, rank: Optional[int] = None, interval: Optional[int] = None): self.rank = rank self.interval = interval @validator('rank') def check_if_rank_in_range(cls, v): """ check if input rank is within range """ if not 0 < v < 1000001: raise HTTPException( status_code=400, detail="Rank Value Must be within range (0,1000000)") return v @validator('interval') def check_if_interval_in_range(cls, v): """ check if input rank is within range """ if not 0 < v < 1000001: raise HTTPException( status_code=400, detail="Interval Value Must be within range (0,1000000)") return v  The FastAPI Endpoint @router.get('fields/',status_code=200) def get_data(params: GetInput = Depends()): if params.rank: result = get_info_by_rank(params.rank) elif params.interval: result = get_info_by_interval(params.interval) return result 

Answer: class GetInput(BaseModel): rank: Optional[int]=None interval: Optional[int]=None @validator("*") def check_range(cls, v): if v: if not 0 < v < 1000001: raise HTTPException(status_code=400, detail="Value Must be within range (0,1000000)") return v   the validator was not working due to not Inheriting the BaseModel Class When the BaseModel Class would get inherited it would throw an error if either of the values is empty thus the additional if statement.
Question: I have a project with some python in it - the python is part of a larger thing - there are several programs in several directories inside a root git - and there is some common code in yet another directory. Running works fine - but pylance in visual studio code sees all of the dependencies as errors, even though most are in the current directory of the script I'm editing. If I open that subdirectory they work fine - but I really want to have visual code open at the larger project level. Is there any way to bung a file somewhere in the directory that tells pylance "here is the python search path" or something?

Answer: You may search for settings about additional import search resolution paths: "python.analysis.extraPaths": [ "path1", "path2", ],  Please have a try. More information view Pylance Settings and Customization.
Question: I'm learning about sessions in Flask and in the documentation it says: "Besides the default client-side based sessions, if you want to handle sessions on the server-side instead, there are several Flask extensions that support this." https://flask.palletsprojects.com/en/2.0.x/quickstart/#sessions What is the different between client-side based sessions and server-side?

Answer:  In addition to the request object there is also a second object called session which allows you to store information specific to a user from one request to the next. This is implemented on top of cookies for you and signs the cookies cryptographically. What this means is that the user could look at the contents of your cookie but not modify it, unless they know the secret key used for signing.  So, the information is physically stored in the cookies, e.g. username=john is the value that's stored in the cookie. That's a "client-side session". The problem with that, as explained above, is that the user can see that data, which is bad if you want to store secret data in the session. If you need that, you need server-side sessions where the data is actually stored server-side, and all the client sees is some random meaningless session id. The session id gets stored in the cookie, and the server looks up the actual session data based on that id somewhere in some database. The advantage of client-side sessions is that the server is entirely stateless, i.e. it doesn't need to store any data itself. That means it doesn't need to do any database lookup to get the data, and you can—for example—run several independent servers in parallel without needing to worry about having a shared session store, which is great for scalability. The advantage of server-side sessions is that you can store more data, as it doesn't need to be sent back and forth with every request, and that the data is not visible to the user.
Question: Let's say I have from enum import Enum class SomeType(Enum): TYPEA = 'type_a' TYPEB = 'type_b' TYPEC = 'type_c'  If I now do SomeType('type_a')  I will get <SomeType.TYPEA: 'type_a'>  as expected. When I do SomeType('type_o')  I will receive  ValueError: 'type_o' is not a valid SomeType  which is also expected. My question is: Can one somehow easily customize the error so that it shows all valid types? So, in my case I would like to have  ValueError: 'type_o' is not a valid SomeType. Valid types are 'type_a', 'type_b', 'type_c'. 

Answer: Use the _missing_ method: from enum import Enum class SomeType(Enum): TYPEA = 'type_a' TYPEB = 'type_b' TYPEC = 'type_c' @classmethod def _missing_(cls, value): raise ValueError( '%r is not a valid %s. Valid types: %s' % ( value, cls.__name__, ', '.join([repr(m.value) for m in cls]), ))  and in use: >>> SomeType('type_a') <SomeType.TYPEA: 'type_a'> >>> SomeType('type_o') ValueError: 'type_o' is not a valid SomeType During handling of the above exception, another exception occurred: Traceback (most recent call last): ... ValueError: 'type_o' is not a valid SomeType. Valid types: 'type_a', 'type_b', 'type_c'  As you see, it's a little clunky with the exception chaining as Enum itself will raise the "primary" ValueError, with your missing error being in the chain. Depending on your needs, you can narrow the ValueError that you are raising in _missing_ to just include the valid types: from enum import Enum class SomeType(Enum): TYPEA = 'type_a' TYPEB = 'type_b' TYPEC = 'type_c' @classmethod def _missing_(cls, value): raise ValueError( 'Valid types: %s' % ( ', '.join([repr(m.value) for m in cls]), )  and in use: >>> SomeType('type_o') ValueError: 'type_o' is not a valid SomeType During handling of the above exception, another exception occurred: Traceback (most recent call last): ... ValueError: Valid types: 'type_a', 'type_b', 'type_c'   Disclosure: I am the author of the Python stdlib Enum, the enum34 backport, and the Advanced Enumeration (aenum) library.
Question: I am developing an application where all the API's are protected by OAuth. I have received the access token from the client, but could not decode and validate the token. I have JWK in the below format { "keys": [ { "kty": "RSA", "x5t#S256": "Some value", "e": "Some Value", "x5t": "Some Value", "kid": "SIGNING_KEY", "x5c": [ "Some Value" ], "key_ops": [ "verify", "encrypt" ], "alg": "RS256", "n": "Some Value" } ] }  How to decode the JWT token using above JWK in Python?

Answer: Fast check of your jwt token https://jwt.io/ otherwise you can try this, but you should know the algorithm used to generate the token (e.g. : HS256) and the key used for signing the token) (e.g. :super_secretkey)  import jwt # pip install pyjwt[crypto] to install the package jwt.decode(token, key='super_secretkey', algorithms=['HS256', ])  Update decode the JWT using JWK import json import jwt #for JWKS that contain multiple JWK public_keys = {} for jwk in jwks['keys']: kid = jwk['kid'] public_keys[kid] = jwt.algorithms.RSAAlgorithm.from_jwk(json.dumps(jwk)) kid = jwt.get_unverified_header(token)['kid'] key = public_keys[kid] payload = jwt.decode(token, key=key, algorithms=['RS256'])
Question: I've recently started using selenium for a project I've been working on for a while that involves automation. One of the roadblocks in the plan was the ReCaptcha system, so I decided to use anti-captcha as the service that would solve the captchas when my bot encountered it. I properly installed the plugin and found some test code with selenium on their site. from python_anticaptcha import AnticaptchaClient, NoCaptchaTaskProxylessTask def captcha_solver(): api_key = 'xxxxxxxxxxxxxxxxxxxxxxx' site_key = '6LdZPw8aAAAAAA_1XrIfloCojPwo4TdJ_A_7ioRy' # grab from site url = 'https://www.rp.gob.pa/' client = AnticaptchaClient(api_key) task = NoCaptchaTaskProxylessTask(url, site_key) job = client.createTask(task) job.join() return job.get_solution_response() captcha = captcha_solver() driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "{}";'.format(captcha)) time.sleep(1) wait.until(EC.element_to_be_clickable((By.XPATH,'//button[@type="submit"]'))).click()  the anticaptcha says the recaptcha is solved, the solved code comes like this 03AGdBq24C2SwOzrdAoGpltxX-8vMuBEjwSmHVIRkVcthtqHEsmm7sEyac1vUgTZQHs7bUtK0YwW6NiduvAmXQt6xVxGRSvO1XhsiRPTfa8spSxRG6scwInLccriAV408I4plNzEykQVQya9v2u4PMyCyrVQ6NADI_A_56DuQvuzhLKuiNL-eN4MvtwEt1ueDefa3nwHUZoW-hgMiEcg1jQ4UhZJ0Ncz1favKF8aMB--Ru1-ewClN41MjyVwREHn1xuCNtnMt5rxaFLt0f5SehaFkdccem1rbCTqsb7lOomTEWpX0TiWKl2kOP9efgOJDlwV84ISncydrQseda7pTlf6nL0m_oUY8U-tnWFQi2i8g_ZWwOgrXb6o9lBapoy0-z0SWZARHKecBbfwHa906mG_b2jh9-IPOI-6rduxTnDw4HDlizXGKOU7Z8Cb8pQAhiaYEejiaBU0X2Dc44dq7CL4Q_365277zoKG4YDwgRXjUstT39e-3C_-lpjdNHMkkz9RJTNe0kOie2i3U-BruAh3trh-vM8F7JU4f8m52F335q3GdUb8FQXL7Fd9hLJpb9KfDMV0pfmRuxl5NoECKRbP2gtTTXUJ0ZwQ  I execute this solved code to g-recaptcha-response textarea and says the selenium to click the button, but the result is this  I cannot solve the recaptcha using anticaptcha, I don't whether my code has a problem, but I followed the official documentation to use the recaptcha. Guys please help me to solve this issue.

Answer: I've finally managed to resolve this myself. In case anyone else is struggling with a similar issue, here was my solution:  Open the console and execute the following cmd: ___grecaptcha_cfg.clients Find the path which has the callback function, in my case it's ___grecaptcha_cfg.clients[0].R.R Use the following code: driver.execute_script(f"___grecaptcha_cfg.clients[0].R.R.callback('{new_token}')") (Remember to change the path accordingly)  Can get the path using the google console  Right click the callback -> copy property path and paste in driver.execute_script and add this on start ___grecaptcha_cfg. and pass the solved token value This Article will help you to find the ___grecaptcha_cfg.clients of your recaptcha site driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "{}";'.format(g_response)) time.sleep(1) driver.execute_script(f"___grecaptcha_cfg.clients[0].R.R.callback('{g_response}')")
Question: I have created a simple jupyter notebook in VS Code and selected it to use my default python3 kernel (/usr/local/bin/python3). Everything works great. Then, I close VS Code and re-open the notebook, it asks me to select the kernel every time. Is there a way to default the kernel of this notebook to my python3 interpreter? In case it helps, when I view the notebook json, it has the following in it:  "kernelspec": { "name": "python3", "display_name": "Python 3.9.6 64-bit" 

Answer: It's not available for now, but they think it is a reasonable request, and considering it. You can refer to this page.
Question: I stumbled upon a weird behaviour: I add a permission to a user object but the permission check fails. permission = Permission.objects.get_by_natural_key(app_label='myapp', codename='my_codename', model='mymodel') user.user_permissions.add(permission) user.has_permission('myapp.my_codename') # this is False!  I found some posts about user permission caching here and here and the solution seems to be to completely reload the object from the database. # Request new instance of User user = get_object_or_404(pk=user_id) # Now note how the permissions have been updated user.has_perms('myapp.my_codename') # now it's True  This seems really overkill for me and very un-django-like. Is there really no way to either clear the permission cache oder reload the foreign keys like you can do for an object with refresh_from_db()? Thanks in advance! Ronny

Answer: You can force the recalculation by deleting the user object's _perm_cache and _user_perm_cache. permission = Permission.objects.get_by_natural_key(app_label='myapp', codename='my_codename', model='mymodel') user.user_permissions.add(permission) user.has_permission('myapp.my_codename') # returns False del user._perm_cache del user._user_perm_cache user.has_permission('myapp.my_codename') # should be True  But this will essentially hit the database again to fetch the updated permissions. Since these are based on internal workings in django and not part of the public API, these cache keys might change in the future, so I would still suggest to just fetch the user again. But that's totally up to you.