eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am unable to download the original ImageNet dataset from their official website. However, I found out that pytorch has ImageNet as one of it’s torch vision datasets. Q1. Is that the original ImageNet dataset? Q2. How do I get the classes for the dataset like it’s being done in Cifar-10 classes = [‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’] 

Answer: The torchvision.datasets.ImageNet is just a class which allows you to work with the ImageNet dataset. You have to download the dataset yourself (e.g. from http://image-net.org/download-images) and pass the path to it as the root argument to the ImageNet class object. Note that the option to download it directly by passing the flag download=True is no longer possible: if download is True: msg = ("The dataset is no longer publicly accessible. You need to " "download the archives externally and place them in the root " "directory.") raise RuntimeError(msg) elif download is False: msg = ("The use of the download flag is deprecated, since the dataset " "is no longer publicly accessible.") warnings.warn(msg, RuntimeWarning)  (source) If you just need to get the class names and the corresponding indices without downloading the whole dataset (e.g. if you are using a pretrained model and want to map the predictions to labels), then you can download them e.g. from here or from this github gist.
Question: The miniforge installer is a relatively new, community-led, minimal conda installer that (as it says in its readme) "can be directly compared to Miniconda, with the added feature that conda-forge is the default channel". It is unclear what is different between miniforge and Miniconda, or what the miniforge use case is. If miniforge is the same as Miniconda except it just uses the conda-forge channel by default, why create a whole different installer - why not just use miniconda and add conda-forge as the first channel to use in ~/.condarc? If miniforge is different from Miniconda, what is different about the two?

Answer: miniforge is the community (conda-forge) driven minimalistic conda installer. Subsequent package installations come thus from conda-forge channel. miniconda is the Anaconda (company) driven minimalistic conda installer. Subsequent package installations come from the anaconda channels (default or otherwise). miniforge started a few months ago because miniconda doens't support aarch64, very quickly the 'PyPy' people jumped on board, and in the mean time there are also miniforge versions for all Linux architectures, as well as MacOS.  Soon there will also be a windows variant (hopefully also for both CPython and PyPy) I guess that an ARMv7 (32Bit ARM) variant is also on the horizon (Raspbian)
Question: I have the following code:  most_important = features_importance_chi(importance_score_tresh, df_user.drop(columns = 'CHURN'),churn) X = df_user.drop(columns = 'CHURN') churn[churn==2] = 1 y = churn # handle undersample problem X,y = handle_undersampe(X,y) # train the model X=X.loc[:,X.columns.isin(most_important)].values y=y.values parameters = { 'application': 'binary', 'objective': 'binary', 'metric': 'auc', 'is_unbalance': 'true', 'boosting': 'gbdt', 'num_leaves': 31, 'feature_fraction': 0.5, 'bagging_fraction': 0.5, 'bagging_freq': 20, 'learning_rate': 0.05, 'verbose': 0 } # split data x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y) train_data = lightgbm.Dataset(x_train, label=y_train) test_data = lightgbm.Dataset(x_test, label=y_test) model = lightgbm.train(parameters, train_data, valid_sets=[train_data, test_data], **feature_name=most_important,** num_boost_round=5000, early_stopping_rounds=100)  and function which returns most_important parameter def features_importance_chi(importance_score_tresh, X, Y): model = ExtraTreesClassifier(n_estimators=10) model.fit(X,Y.values.ravel()) feature_list = pd.Series(model.feature_importances_, index=X.columns) feature_list = feature_list[feature_list > importance_score_tresh] feature_list = feature_list.index.values.tolist() return feature_list  Funny thing is that this code in Spyder returns the following error LightGBMError: Do not support special JSON characters in feature name.  but in jupyter works fine. I am able to print the list of most important features. Any idea what could be the reason for this error?

Answer: You know what, this message is often found on LGBMClassifier () models, i.e. LGBM. Simply drop this line at the beginning as soon as you upload the data from the pandas and you have a problem with your head: import re df = df.rename(columns = lambda x:re.sub('[^A-Za-z0-9_]+', '', x))
Question: According to poetry's docs, the proper way to setup a new project is with poetry new poetry-demo, however this creates a project based on the now deprecated python2.7 by creating the following toml file: [tool.poetry] name = "poetry-demo" version = "0.1.0" description = "" authors = ["Harsha Goli <harshagoli@gmail.com>"] [tool.poetry.dependencies] python = "^2.7" [tool.poetry.dev-dependencies] pytest = "^4.6" [build-system] requires = ["poetry>=0.12"] build-backend = "poetry.masonry.api"  How can I update this to 3.7? Simply changing python = "^2.7" to python = "^3.7" results in the following error when poetry install is run: [SolverProblemError] The current project's Python requirement (2.7.17) is not compatible with some of the required packages Python requirement: - zipp requires Python >=3.6 Because no versions of pytest match >=4.6,<4.6.9 || >4.6.9,<5.0 and pytest (4.6.9) depends on importlib-metadata (>=0.12), pytest (>=4.6,<5.0) requires importlib-metadata (>=0.12). And because no versions of importlib-metadata match >=0.12,<1.5.0 || >1.5.0 and importlib-metadata (1.5.0) depends on zipp (>=0.5), pytest (>=4.6,<5.0) requires zipp (>=0.5). Because zipp (3.1.0) requires Python >=3.6 and no versions of zipp match >=0.5,<3.1.0 || >3.1.0, zipp is forbidden. Thus, pytest is forbidden. So, because poetry-demo depends on pytest (^4.6), version solving failed. 

Answer: Interestingly, poetry is silently failing due to a missing package the tool itself relies on and continues to install a broken venv. Here's how you fix it. sudo apt install python3-venv poetry env remove python3 poetry install  I had to remove pytest, and then reinstall with poetry add pytest. EDIT: I ran into this issue again when upgrading a project from python3.7 to python3.8 - for this instead of installing python3-venv, you'd want to install python3.8-venv instead
Question: OK, so somehow I have mangled my python3 installation under macOS Mojave and I'm not sure how. I've used macports for years to keep python up to date but when I installed python38 now I cannot run python3 at all. I always get this: $ python3.8 Fatal Python error: config_get_locale_encoding: failed to get the locale encoding: nl_langinfo(CODESET) failed Python runtime state: preinitialized $  I uninstalled the macports version and reinstalled, same thing. Uninstalled and then installed fresh from python.org, same thing. python27 runs fine. python37 also runs fine. python38 won't even work if I use $python3.8 -I so it's not some site package weirdness. Here's the really weird bit: while I cannot run python38 from a shell (any shell, tried from bash , I can launch python38 from the GUI using IDLE.app.  Oddly, on my other machine (my laptop), python38 installed with macports works just fine.  I'm flummoxed and I don't flummox easily. Any ideas?

Answer: Try setting LANG with a locale: export LANG="en_US.UTF-8"
Question: the exact error I get is : flask.cli.NoAppException: While importing "application", an ImportError was raised:Traceback (most recent call last): File "/home/harshit/.local/lib/python3.6/site-packages/flask/cli.py", line 240, in locate_app __import__(module_name) File "/home/harshit/Documents/project1/application.py", line 18, in <module> Session(app) File "/home/harshit/.local/lib/python3.6/site-packages/flask_session/__init__.py", line 54, in __init__ self.init_app(app) File "/home/harshit/.local/lib/python3.6/site-packages/flask_session/__init__.py", line 61, in init_app app.session_interface = self._get_interface(app) File "/home/harshit/.local/lib/python3.6/site-packages/flask_session/__init__.py", line 93, in _get_interface config['SESSION_USE_SIGNER'], config['SESSION_PERMANENT']) File "/home/harshit/.local/lib/python3.6/site-packages/flask_session/sessions.py", line 313, in __init__ from werkzeug.contrib.cache import FileSystemCache ModuleNotFoundError: No module named 'werkzeug.contrib'  I am trying to import sessions from Flask

Answer: Werkzeug 1.0.0 has removed deprecated code, including all of werkzeug.contrib. You should use alternative libraries for new projects. werkzeug.contrib.session was extracted to secure-cookie.  If an existing project you're using needs something from contrib, you'll need to downgrade to Werkzeug<1: pip3 install Werkzeug<1
Question: Even if a class is inherited from ABC, it can still be instantiated unless it contains abstract methods. Having the code below, what is the best way to prevent an Identifier object from being created: Identifier(['get', 'Name'])? from abc import ABC from typing import List from dataclasses import dataclass @dataclass class Identifier(ABC): sub_tokens: List[str] @staticmethod def from_sub_tokens(sub_tokens): return SimpleIdentifier(sub_tokens) if len(sub_tokens) == 1 else CompoundIdentifier(sub_tokens) @dataclass class SimpleIdentifier(Identifier): pass @dataclass class CompoundIdentifier(Identifier): pass 

Answer: You can create a AbstractDataclass class which guarantees this behaviour, and you can use this every time you have a situation like the one you described. @dataclass class AbstractDataclass(ABC): def __new__(cls, *args, **kwargs): if cls == AbstractDataclass or cls.__bases__[0] == AbstractDataclass: raise TypeError("Cannot instantiate abstract class.") return super().__new__(cls)  So, if Identifier inherits from AbstractDataclass instead of from ABC directly, modifying the __post_init__ will not be needed. @dataclass class Identifier(AbstractDataclass): sub_tokens: List[str] @staticmethod def from_sub_tokens(sub_tokens): return SimpleIdentifier(sub_tokens) if len(sub_tokens) == 1 else CompoundIdentifier(sub_tokens) @dataclass class SimpleIdentifier(Identifier): pass @dataclass class CompoundIdentifier(Identifier): pass  Instantiating Identifier will raise TypeError but not instantiating SimpleIdentifier or CompountIdentifier. And the AbstractDataclass can be re-used in other parts of the code.
Question: Currently, plotly express treemap shows only label inside treemap. How to include the value alongside the label? 

Answer: That's why I don't like express, there are too many limitations and to make these kinds of changes you have to access the trace either way. From my point of view it is better and more code-transparent to use plain plotly instead. That being said, you can access the textinfo attribute of the trace to do this. From the reference:  Determines which trace information appear on the graph. Any combination of "label", "text", "value", "current path", "percent root", "percent entry", "percent parent" joined with a "+" OR "none".  Taking an example from the site: df = px.data.tips() fig = px.treemap(df, path=['day', 'time', 'sex'], values='total_bill') # this is what I don't like, accessing traces like this fig.data[0].textinfo = 'label+text+value+current path' fig.layout.hovermode = False fig.show()   Also take a look at the texttemplate attribute for formatting options.
Question: Do you know any elegant way to do inference on 2 python processes with 1 GPU tensorflow? Suppose I have 2 processes, first one is classifying cats/dogs, 2nd one is classifying birds/planes, each process is running different tensorflow model and run on GPU. These 2 models will be given images from different cameras continuously. Usually, tensorflow will occupy all memory of the entire GPU. So when you start another process, it will crash saying OUT OF MEMORY or failed convolution CUDA or something along that line. Is there a tutorial/article/sample code that shows how to load 2 models in different processes and both run in parallel? This is very useful also in case you are running a model inference while you are doing some heavy graphics e.g. playing games. I also want to know how running the model affects the game. I've tried using python Thread and it works but each model predicts 2 times slower (and you know that python thread is not utilizing multiple CPU cores). I want to use python Process but it's not working. If you have sample few lines of code that work I would appreciate that very much. I've attached current Thread code also: 

Answer: OK. I think I've found the solution now. I use tensorflow 2 and there are essentially 2 methods to manage the memory usage of GPU.  set memory growth to true set memory limit to some number  You can use both methods, ignore all the warning messages about out of memory stuff. I still don't know what it exactly means but the model is still running and that's what I care about. I measured the exact time the model uses to run and it's a lot better than running on CPU. If I run both processes at the same time, the speed drop a bit, but it's still lot better than running on CPU. For memory growth approach, my GPU is 3GB so first process try to allocate everything and then 2nd process said out of memory. But it still works. For memory limit approach, I set the limit to some number e.g. 1024 MB. Both processes work.  So What is the right minimum number that you can set?  I tried reducing the memory limit until I found that my model works with 64 MB limit fine. The prediction speed is still the same as when I set the memory limit to 1024 MB. When I set the memory limit to 32MB, I noticed 50% speed drop. When I set to 16 MB, the model refuses to run because it does not have enough memory to store the image tensor. This means that my model requires minimum of 64 MB which is very little considering that I have 3GB to spare. This also allows me to run the model while playing some video games. Conclusion: I chose to use the memory limit approach with 64 MB limit. You can check how to use memory limit here: https://www.tensorflow.org/guide/gpu I suggest you to try changing the memory limit to see the minimum you need for your model. You will see speed drop or model refusing to run when the memory is not enough.
Question: Why are future imports limited to only certain functionality? Is there no way to get the walrus operator in Python 3.7? I thought this would work, but it doesn't: from __future__ import walrus  It doesn't work because walrus isn't in the list of supported features: __future__.all_feature_names ['nested_scopes', 'generators', 'division', 'absolute_import', 'with_statement', 'print_function', 'unicode_literals', 'barry_as_FLUFL', 'generator_stop', 'annotations']  Are there any other alternatives short of using python 3.8?

Answer: If the version of Python you're using doesn't contain an implementation of a feature, then you cannot use that feature; writing from __future__ import ... cannot cause that feature to be implemented in the version of Python you have installed. The purpose of __future__ imports is to allow an "opt-in" period for new features which could break existing programs. For example, when the / operator's behaviour on integers was changed so that 3/2 was 1.5 instead of 1 (i.e. floor division), this would have broken a lot of code if it was just changed overnight. So both behaviours were implemented in the next few versions of Python, and if you were using one of those newer versions then you could choose the new behaviour with from __future__ import division. But you were only able to do so because the version of Python you were using did implement the new behaviour. The walrus operator was introduced in Python 3.8, so if you are using a version prior to 3.8 then it does not contain an implementation of that operator, so you cannot use it. There was no need to use __future__ to make the walrus operator "opt-in", since the introduction of a new operator with new syntax could not have broken any existing code.
Question: Forgive me if this question has been asked before but I could not find any related answer. Consider a function that takes a numerical type as input parameter: def foo(a): return ((a+1)*2)**4;  This works with integers, floats and complex numbers. Is there a basic type so that I can do a type hinting (of a real existing type/base class), such as: def foo(a: numeric): return ((a+1)*2)**4;  Furthermore I need to use this in a collection type parameter, such as: from typing import Collection; def foo(_in: Collection[numeric]): return ((_in[0]+_in[1])*2)**4; 

Answer: PEP 3141 added abstract base classes for numbers, so you could use: from numbers import Number def foo(a: Number) -> Number: ...
Question: Am using tensorflow 2.1 on Windows 10. On running model.add(Conv3D(16, (22, 5, 5), strides=(1, 2, 2), padding='valid',activation='relu',data_format= "channels_first", input_shape=input_shape))  on spyder, I get the this error: { AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'experimental_list_devices' }  How can I solve this error?

Answer: I found the answer here - https://github.com/keras-team/keras/issues/13684. I had the same issue for load_model() from keras under Anaconda:  AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'experimental_list_devices'  I found source of problem in   ...\anaconda3\envs\tf_env\Lib\site-packages\keras\backend\tensorflow_backend.py  In line 506 I changed line _LOCAL_DEVICES = tf.config.experimental_list_devices()  to devices = tf.config.list_logical_devices() _LOCAL_DEVICES = [x.name for x in devices]  And it works
Question: I'm working on a text classification problem (e.g. sentiment analysis), where I need to classify a text string into one of five classes. I just started using the Huggingface Transformer package and BERT with PyTorch. What I need is a classifier with a softmax layer on top so that I can do 5-way classification. Confusingly, there seem to be two relevant options in the Transformer package: BertForSequenceClassification and BertForMultipleChoice. Which one should I use for my 5-way classification task? What are the appropriate use cases for them? The documentation for BertForSequenceClassification doesn't mention softmax at all, although it does mention cross-entropy. I am not sure if this class is only for 2-class classification (i.e. logistic regression).  Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.  labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).   The documentation for BertForMultipleChoice mentions softmax, but the way the labels are described, it sound like this class is for multi-label classification (that is, a binary classification for multiple labels).  Bert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.  labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices] where num_choices is the size of the second dimension of the input tensors.   Thank you for any help.

Answer: The answer to this lies in the (admittedly very brief) description of what the tasks are about:  [BertForMultipleChoice] [...], e.g. for RocStories/SWAG tasks.  When looking at the paper for SWAG, it seems that the task is actually learning to choose from varying options. This is in contrast to your "classical" classification task, in which the "choices" (i.e., classes) do not vary across your samples, which is exactly what BertForSequenceClassification is for. Both variants can in fact be for an arbitrary number of classes (in the case of BertForSequenceClassification), respectively choices (for BertForMultipleChoice), via changing the labels parameter in the config. But, since it seems like you are dealing with a case of "classical classification", I suggest using the BertForSequenceClassification model. Shortly addressing the missing Softmax in BertForSequenceClassification: Since classification tasks can compute loss across classes indipendent of the sample (unlike multiple choice, where your distribution is changing), this allows you to use Cross-Entropy Loss, which factors in Softmax in the backpropagation step for increased numerical stability.
Question: I'm trying to use rolling and apply function to print window but I got the error says  File "pandas/_libs/window.pyx", line 1649, in pandas._libs.window.roll_generic TypeError: must be real number, not NoneType  My code is following  def print_window(window): print(window) print('==================') def example(): df = pd.read_csv('window_example.csv') df.rolling(5).apply(print_window)  My data is like  number sum mean 1 1 1 2 3 1.5 3 6 2 4 10 2.5 5 15 3 6 20 4  How should I slove this error? I didn't find similar questions on this error  Thanks !

Answer: This behavior appeared in pandas=1.0.0. The function of the apply is now expected to return a single value to affect the corresponding column with.  https://pandas.pydata.org/pandas-docs/version/1.0.0/reference/api/pandas.core.window.rolling.Rolling.apply.html#pandas.core.window.rolling.Rolling.apply A workaround for your code would be : def print_window(window): print(window) print('==================') return 0 def example(): df = pd.read_csv('window_example.csv') df.rolling(5).apply(print_window)
Question: Today, someone asked me about static methods and said is that right that static method can't access or modify a class state?

Answer: General OO answer: the state of an object is the values of it's attributes. For example, given  class Point: def __init__(self, x, y): self.x = x self.y = y p = Point(42, 43)  then the state of p is {"x": 42, "y": 43} To modify an object's state, a method need to have access to this object. For ordinary methods, this is provided by the self parameter. Now Python's classes are objects too (instance of the type class), so Python has "classmethods" which can be invoked on either an instance or the class itself, but get the class object itself instead of an instance. Those classmethods can then modify the class's state (class attributes, which are shared by all instances of the class). A staticmethod gets neither the instance nor class, so it can't change neither the class nor instance's state indeed.