eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I use Fast API to create a web service. There are following sqlAlchemy models: class User(Base): __tablename__ = 'user' account_name = Column(String, primary_key=True, index=True, unique=True) email = Column(String, unique=True, index=True, nullable=False) roles = relationship("UserRole", back_populates="users", lazy=False, uselist=True) class UserRole(Base): __tablename__ = 'user_role' __table_args__ = (UniqueConstraint('role_name', 'user_name', name='user_role_uc'),) role_name = Column(String, ForeignKey('role.name'), primary_key=True) user_name = Column(String, ForeignKey('user.account_name'), primary_key=True) users = relationship("User", back_populates="roles")  Pydantic schemas are below: class UserRole(BaseModel): role_name: str class Config: orm_mode = True class UserBase(BaseModel): account_name: str email: EmailStr roles: List[UserRole] = [] class Config: orm_mode = True  What I have now is: { "account_name": "Test.Test", "email": "Test.Test@test.com", "roles": [ { "role_name": "all:admin" }, { "role_name": "all:read" } ] }  What I want to achieve is to get user from api in following structure: { "account_name": "Test.Test", "email": "Test.Test@test.com", "roles": [ "all:admin", "all:read" ] }  Is that possible? How should I change schemas to get this?

Answer: If you are okay with handling the how to "get user from api" problem statement by modifying the fastapi path definition, see below. Can you change the response model used by the fastapi path definition in order to handle the desired output format? Example pydantic response model definition: class UserResponse(BaseModel): account_name: str email: EmailStr roles: List[str]  Example sqlalchemy query + serialization function: def get_user_response(user_id) -> UserResponse: user = User.query.get(user_id) user_roles = UserRole.query.filter(user=user_id).all() role_names = [r.role_name for r in user_roles] response = UserResponse( account_name=user.account_name, email=user.email, roles=role_names } return response  Example fastapi path definition: @app.get("/users/{user_id}", response_model=UserResponse) async def read_item(user_id): return get_user_response(user_id)  Considerations:  I'm using a user_id for the user queries but this can be replaced with whatever you end up using as your primary key for that table. The UserResponse response model is very similar to UserBase (you could potentially subclass UserBase instead of model to avoid the redefinition of account_name and email, with the tradeoff of having to override the class' Config). There may be a way to override the serialization format of the UserBase sqlalchemy model object that gets automatically serialized when you query the model from the database and allows you to eliminate or reduce the code in the get_user_response() function in the example definition above.
Question: I am a bit confused by the PEP257 standard for documenting classes. It says, "The docstring for a class should summarize its behavior and list the public methods and instance variables" But it also says that all functions should have dosctrings (which, of course, I want, so that help() works). But this seems to involve duplication i.e. class foo: """A class Attributes ---------- bar : str A string Methods ------- __init__(fish): Constructor, fish is a str which self.bar will be set to. baz(): A function which does blah """ def __init__(self, fish): """ Constructs an XRTProductRequest object. Parameters ---------- fish : str A string, which the self.bar attribute will be set to. """ etc...  This then is rather error prone because it means that when I realise that __init__ also needs to recieve an int, then I have to remember to update the docs in 2 places, which I can guarantee I will forget. It also makes the pydoc output duplicated: it prints my class docstring, but then says, "Methods defined here" and goes on to list all of the methods, via their own docstrings. So, is this duplication really part of PEP257, or am I mis-reading it? Should I drop the "Methods"section of the class docstring, since each method has its own docstring? Or is this duplication really part of the standard? TIA

Answer: Yes just drop the methods section from the class docstring. I've never ever seen something like that used.(It is used in few places in the standard library.) The class docstring needs to just describe the class and the docstring of individual methods then handle describing themselves. Also the wording in the PEP to me means that the class docstring "should" list the public methods, but not describe them in any other way.(This is also how the above standard library example does it.) But as said, I would never even do that, since the code speaks for itself and that kind of listing is bound to get out-of-date. Final note: I personally prefer to use the Google docstring style, because to me it's the clearest and cleanest.
Question: For example, if I want to solve the MNIST classification problem, we have 10 output classes. With PyTorch, I would like to use the torch.nn.CrossEntropyLoss function. Do I have to format the targets so that they are one-hot encoded or can I simply use their class labels that come with the dataset?

Answer: nn.CrossEntropyLoss expects integer labels. What it does internally is that it doesn't end up one-hot encoding the class label at all, but uses the label to index into the output probability vector to calculate the loss should you decide to use this class as the final label. This small but important detail makes computing the loss easier and is the equivalent operation to performing one-hot encoding, measuring the output loss per output neuron as every value in the output layer would be zero with the exception of the neuron indexed at the target class. Therefore, there's no need to one-hot encode your data if you have the labels already provided. The documentation has some more insight on this: https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html. In the documentation you'll see targets which serves as part of the input parameters. These are your labels and they are described as:  This clearly shows how the input should be shaped and what is expected. If you in fact wanted to one-hot encode your data, you would need to use torch.nn.functional.one_hot. To best replicate what the cross entropy loss is doing under the hood, you'd also need nn.functional.log_softmax as the final output and you'd have to additionally write your own loss layer since none of the PyTorch layers use log softmax inputs and one-hot encoded targets. However, nn.CrossEntropyLoss combines both of these operations together and is preferred if your outputs are simply class labels so there is no need to do the conversion.
Question: I'm currently developing a service in Django which makes use of a slow external API (takes about 10s to get a response), which means the connections to my server are kept open waiting for the external API to respond, and occupying worker time/resources. I know I can use gunicorn's thread or gevent workers to add concurrency, but can't seem to grasp the exact difference between using gunicorn with gevent workers and uvicorn (or any other server) with the asgi interface. What would be the criteria for using one over the other? Django still doesn't fully support async/await views. Would it be better if I just stick with gevent workers?

Answer: Gunicorn has a pre-fork worker model A pre-fork worker model basically means a master creates forks which handle each request. A fork is a completely separate *nix process (Source). Uvicorn is a ASGI server running uvloop Python async needs a event loop for it to use it's async features. And uvloop is an alternative to the asyncio loop. So if you're having async calls in your code you could use uvloop internally or use uvicorn as your ASGI server. Note: You still have just a single event loop. You can choose to use gunicorn and uvicorn to have multiple workers and use uvloop. In your django application it makes sense to use uvloop internally (if you choose to) and use gunicorn as your ASGI.
Question: Is there a way to convert a pydantic model to query parameters in fastapi?  Some of my endpoints pass parameters via the body, but some others pass them directly in the query. All this endpoints share the same data model, for example:  class Model(BaseModel): x: str y: str  I would like to avoid duplicating my definition of this model in the definition of my "query-parameters endpoints", like for example test_query in this code: class Model(BaseModel): x: str y: str @app.post("/test-body") def test_body(model: Model): pass @app.post("/test-query-params") def test_query(x: str, y: str): pass  What's the cleanest way of doing this?

Answer: The documentation gives a shortcut to avoid this kind of repetitions. In this case, it would give: from fastapi import Depends @app.post("/test-query-params") def test_query(model: Model = Depends()): pass  This will allow you to request /test-query-params?x=1&y=2 and will also produce the correct OpenAPI description for this endpoint. Similar solutions can be used for using Pydantic models as form-data descriptors.
Question: I made a simple CLI using Typer and Pillow to change image opacity and this program only have one option: opacity. But when I run python opacity.py --help it gives me the two typerCLI options: Options: --install-completion [bash|zsh|fish|powershell|pwsh] Install completion for the specified shell. --show-completion [bash|zsh|fish|powershell|pwsh] Show completion for the specified shell, to copy it or customize the installation. --help Show this message and exit.  There's a way to disable it? I didn't find on docs.

Answer: I met the same problem today, i couldn't find anything except this question so dived in the source to find how Typer automatically adds this line in app, so i found this, when Typer initialiazing itself it automatically sets add_completion to True class Typer: def __init__(add_completion: bool = True)  So when you initialiazing your app you can add this app = typer.Typer(add_completion=False)  This is how it looks after you add add_completion=False Usage: main.py [OPTIONS] COMMAND [ARGS]... Options: --help Show this message and exit.
Question: I am still learning tensorflow and keras, and I suspect this question has a very easy answer I'm just missing due to lack of familiarity. I have a PrefetchDataset object: > print(tf_test) $ <PrefetchDataset shapes: ((None, 99), (None,)), types: (tf.float32, tf.int64)>  ...made up of features and a target. I can iterate over it using a for loop: > for example in tf_test: > print(example[0].numpy()) > print(example[1].numpy()) > exit() $ [[-0.31 -0.94 -1.12 ... 0.18 -0.27] [-0.22 -0.54 -0.14 ... 0.33 -0.55] [-0.60 -0.02 -1.41 ... 0.21 -0.63] ... [-0.03 -0.91 -0.12 ... 0.77 -0.23] [-0.76 -1.48 -0.15 ... 0.38 -0.35] [-0.55 -0.08 -0.69 ... 0.44 -0.36]] [0 0 1 0 1 0 0 0 1 0 1 1 0 1 0 0 0 ... 0 1 1 0]  However, this is very slow. What I'd like to do is access the tensor corresponding to the class labels and turn that into a numpy array, or a list, or any sort of iterable that can be fed into scikit-learn's classification report and/or confusion matrix: > y_pred = model.predict(tf_test) > print(y_pred) $ [[0.01] [0.14] [0.00] ... [0.32] [0.03] [0.00]] > y_pred_list = [int(x[0]) for x in y_pred] # assumes value >= 0.5 is positive prediction > y_true = [] # what I need help with > print(sklearn.metrics.confusion_matrix(y_true, y_pred_list)  ...OR access the data such that it could be used in tensorflow's confusion matrix: > labels = [] # what I need help with > predictions = y_pred_list # could we just use a tensor? > print(tf.math.confusion_matrix(labels, predictions)  In both cases, the general ability to grab the target data from the original object in a manner that isn't computationally expensive would be very helpful (and might help with my underlying intuitions re: tensorflow and keras). Any advice would be greatly appreciated.

Answer: You can convert it to a list with list(ds) and then recompile it as a normal Dataset with tf.data.Dataset.from_tensor_slices(list(ds)). From there your nightmare begins again but at least it's a nightmare that other people have had before. Note that for more complex datasets (e.g. nested dictionaries) you will need more preprocessing after calling list(ds), but this should work for the example you asked about. This is far from a satisfying answer but unfortunately the class is entirely undocumented and none of the standard Dataset tricks work.
Question: I'm resampling my data (multiclass) by using SMOTE.  sm = SMOTE(random_state=1) X_res, Y_res = sm.fit_resample(X_train, Y_train)  However, I'm getting this attribute error. Can anyone help?

Answer: Short answer You need to upgrade scikit-learn to version 0.23.1. Long answer The newest version 0.7.0 of imbalanced-learn seems to have an undocumented dependency on scikit-learn v0.23.1. It would give you AttributeError: 'SMOTE' object has no attribute '_validate_data' if your scikit-learnis 0.22 or below. If you are using Anaconda, installing scikit-learn version 0.23.1 might be tricky. conda update scikit-learn might not update scikit-learn version 0.23 or higher because the newest scikit-learn version Conda has at this point of time is 0.22.1. If you try to install it using conda install scikit-learn=0.23.1 or pip install scikit-learn==0.23.1, you will get tons of compatibility checks and installation might not be quick. Therefore the easiest way to install scikit-learn version 0.23.1 in Anaconda is to create a new virtual environment with minimum packages so that there are less or no conflict issues. Then, in the new virtual environment install scikit-learn version 0.23.1 followed by version 0.7.0 of imbalanced-learn. conda create -n test python=3.7.6 conda activate test pip install scikit-learn==0.23.1 pip install imbalanced-learn==0.7.0  Finally, you need to reinstall your IDE in the new virtual environment in order to use these packages. However, once scikit-learn version 0.23.1 becomes available in Conda and there are no compatibility issues, you can install it in the base environment directly.
Question: I have an abstract base class in Python which defines an abstract method. I want to decorate it with a timer function such that every class extending and implementing this base class is timed and doesn't need to be manually annotated. Here's what I have import functools import time import abc class Test(metaclass=abc.ABCMeta): @classmethod def __subclasshook__(cls, subclass): return (hasattr(subclass, 'apply') and callable(subclass.apply)) @abc.abstractmethod def apply(self, a: str) -> str: raise NotImplementedError def timer(func): @functools.wraps(func) def wrapper_timer(*args, **kwargs): start_time = time.perf_counter() value = func(*args, **kwargs) end_time = time.perf_counter() run_time = end_time - start_time print(f"Finished {func.__name__!r} in {run_time:.4f} secs") return value return wrapper_timer def __getattribute__(self, name): if name == "apply": func = getattr(type(self), "apply") return self.timer(func) return object.__getattribute__(self, name) class T2(Test): def apply(self, a: str) -> str: return a if __name__ == '__main__': t = T2() t.apply('a')  The error I get is as follow Traceback (most recent call last): File "/Users/blah/test.py", line 41, in <module> t.apply('a') File "/Users/blah/test.py", line 20, in wrapper_timer value = func(*args, **kwargs) TypeError: apply() missing 1 required positional argument: 'a'  I think understand the error python thinks that the apply method of the T2() object is a classmethod however I am not sure why given that I call getattr(type(self), "apply"). Is there a way to get the instance method?

Answer: Use __init_subclass__ to apply the timer decorator for you. (timer, by the way, doesn't need to be defined in the class; it's more general than that.) __init_subclass__ is also a more appropriate place to determine if apply is callable. import abc import functools import time def timer(func): @functools.wraps(func) def wrapper_timer(*args, **kwargs): start_time = time.perf_counter() value = func(*args, **kwargs) end_time = time.perf_counter() run_time = end_time - start_time print(f"Finished {func.__name__!r} in {run_time:.4f} secs") return value return wrapper_timer class Test(metaclass=abc.ABCMeta): def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) # ABCMeta doesn't let us get this far if cls.apply isn't defined if not callable(cls.apply): raise TypeError("apply not callable") cls.apply = timer(cls.apply) @abc.abstractmethod def apply(self, a: str) -> str: raise NotImplementedError class T2(Test): def apply(self, a: str) -> str: return a if __name__ == '__main__': t = T2() t.apply('a')
Question: I need to create a plot on plotly with a newline. Preferably the text on the top is larger than the second line, but not necessary. fig.update_layout( title=go.layout.Title( text=title, xref="paper", x=0.5, ),  For the title I have tried title = "Hello \n World" title = "Hello" + "\n" + "World" title = "$Hello \\ World$" title = "$Hello \newline World$"  All run without error, but the newline is ignored. I am not sure how to implement this. Thank you for the help.

Answer: You can add a <br> tag to the title and it will be interpreted as an HTML line break. This is supported by this doc. You can see the result in the following example: from dash import Dash import dash_core_components as dcc import dash_html_components as html external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css'] app = Dash(__name__, external_stylesheets=external_stylesheets) app.layout = html.Div(children=[ dcc.Graph( id='example-graph', figure={ 'data': [ {'x': [1, 2, 3], 'y': [4, 1, 2], 'type': 'bar', 'name': 'SF'}, {'x': [1, 2, 3], 'y': [2, 4, 5], 'type': 'bar', 'name': u'Montréal'}, ], 'layout': { 'title': 'Dash Data <br> Visualization' } } ) ]) if __name__ == '__main__': app.run_server(debug=True)
Question: With a simple custom exception class defined as: class MyError(Exception): pass  And this call: foo = 'Some more info' raise MyError("%s: there was an error", foo)  pylint gives:  Exception arguments suggest string formatting might be intended pylint(raising-format-tuple)  What does this message mean?

Answer: Any one of these fixes the message, depending on your version of Python. foo = 'Some more info' raise MyError("%s: there was an error" % foo ) raise MyError("{}: there was an error".format(foo)) raise MyError(f"{foo}: there was an error")  The message is triggered when pylint sees the %s tag in the string with no following arguments. Instead of raising an exception with the string "Some more info: there was an error" you'll get an exception with a tuple, where the first element is ": there was an error" and the second is the contents of foo. This is likely not the intended effect. In the code I was using, there was extensive use of logging, and I suspect the original author confused the exception raising with lazy logging.
Question: I just learned about flake8, which calls itself "Flake8: Your Tool For Style Guide Enforcement." While flake8 will find many Python whitespace errors and enforce PEP8, it does not appear to have an option to automatically fix problematic python code. autopep8 does appear to have this option (called --in-place), but flake8 seems to have much wider support. Is there a way to make flake8 fix my code?

Answer: no, flake8 is a linter only -- that is, it only checks your code. (technically, flake8 doesn't even check your code -- it is just a framework for other linters to plug into and provides inclusion / exclusion / etc. on top of other tools) if you want something which fixes your code, you'll want a code formatter to do code formatting (such as autopep8 / add-trailing-comma / yapf / black / etc.)
Question: I have two large NumPy arrays each with shape of (519990,) that look something like this: Order = array([0, 0, 0, 5, 6, 10, 14, 14, 14, 23, 23, 39]) Letters = array([A, B, C, D, E, F, G, H, I, J, K, L])  As you can see the first array is always in ascending and a positive number. I would like to group everything within the Letters to Order to turn out looking like this: {0:[A,B,C], 5:[D], 6:[E], 10:[F], 14:[G, H, I], 23:[J, K], 39:[L]}  The code I have to do this is: df = pd.DataFrame() df['order'] = Order df['letters'] = Letters linearDict = df.grouby('order').apply(lambda dfg:dfg.drop('order', axis=1).to_dict(orient='list')).to_dict() endProduct = {} for k, v in linearDict.items(): endProduct[k] = np.array(linearDict[k]['letter'][0:]) enProduct = {0:array([A,B,C]), 5:array([D]), 6:array([E]), 10:array([F]), 14:array([G, H, I]), 23:array([J, K]), 39:array([L])}  My problem is this process is BEYOND slow. It's such a drain on the system that it causes my Jupyter Notebook to crash. Is there a faster way of doing this?

Answer: We could leverage the fact that Order is sorted, to simply slice Letters after getting the intervaled-indices, like so - def numpy_slice(Order, Letters): Order = np.asarray(Order) Letters = np.asarray(Letters) idx = np.flatnonzero(np.r_[True,Order[:-1]!=Order[1:],True]) return {Order[i]:Letters[i:j] for (i,j) in zip(idx[:-1],idx[1:])}  Sample run - In [66]: Order Out[66]: array([16, 16, 16, 16, 23, 30, 33, 33, 39, 39, 39, 39, 39, 39, 39]) In [67]: Letters Out[67]: array(['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O'], dtype='<U1') In [68]: numpy_slice(Order, Letters) Out[68]: {16: array(['A', 'B', 'C', 'D'], dtype='<U1'), 23: array(['E'], dtype='<U1'), 30: array(['F'], dtype='<U1'), 33: array(['G', 'H'], dtype='<U1'), 39: array(['I', 'J', 'K', 'L', 'M', 'N', 'O'], dtype='<U1')}
Question: I am trying to display an image stored as a pytorch tensor. trainset = datasets.ImageFolder('data/Cat_Dog_data/train/', transform=transforms) trainload = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True) images, labels = iter(trainload).next() image = images[0] image.shape >>> torch.Size([3, 224, 224]) # pyplot doesn't like this, so reshape image = image.reshape(224,224,3) plt.imshow(image.numpy())  This method is displaying a 3 by 3 grid of the same image, always in greyscale. For example:  How do I fix this so that the single color image is displayed correctly?

Answer: That's very odd. Try putting the channels last by permuting rather than reshaping: image.permute(1, 2, 0)