eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Given a sentiment classification dataset, I want to fine-tune Bert. As you know that BERT created to predict the next sentence given the current sentence. Thus, to make the network aware of this, they inserted a [CLS] token in the beginning of the first sentence then they add [SEP] token to separate the first from the second sentence and finally another [SEP] at the end of the second sentence (it's not clear to me why they append another token at the end). Anyway, for text classification, what I noticed in some of the examples online (see BERT in Keras with Tensorflow hub) is that they add [CLS] token and then the sentence and at the end another [SEP] token. Where in other research works (e.g. Enriching Pre-trained Language Model with Entity Information for Relation Classification) they remove the last [SEP] token. Why is it/not beneficial to add the [SEP] token at the end of the input text when my task uses only single sentence?

Answer: Im not quite sure why BERT needs the separation token [SEP] at the end for single-sentence tasks, but my guess is that BERT is an autoencoding model that, as mentioned, originally was designed for Language Modelling and Next Sentence Prediction. So BERT was trained that way to always expect the [SEP] token, which means that the token is involved in the underlying knowledge that BERT built up during training. Downstream tasks that followed later, such as single-sentence use-cases (e.g. text classification), turned out to work too with BERT, however the [SEP] was left as a relict for BERT to work properly and thus is needed even for these tasks. BERT might learn faster, if [SEP] is appended at the end of a single sentence, because it encodes somewhat of a knowledge in that token, that this marks the end of the input. Without it, BERT would still know where the sentence ends (due to the padding tokens), which explains that fore mentioned research leaves away the token, but this might slow down training slightly, since BERT might be able to learn faster with appended [SEP] token, especially if there are no padding tokens in a truncated input.
Question: let me take a 2D matrix as example: mat = torch.arange(9).view(3, -1) tensor([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) torch.sum(mat, dim=-2) tensor([ 9, 12, 15])  I find the result of torch.sum(mat, dim=-2) is equal to torch.sum(mat, dim=0) and dim=-1 equal to dim=1. My question is how to understand the negative dimension here. What if the input matrix has 3 or more dimensions?

Answer: The minus essentially means you go backwards through the dimensions. Let A be a n-dimensional matrix. Then dim=n-1=-1, dim=n-2=-2, ..., dim=1=-(n-1), dim=0=-n. See the numpy doc for more information, as pytorch is heavily based on numpy.
Question: I'm using SVC from sklearn.svm for binary classification in python. For the gamma parameter it says that it's default value is  . I'm having a hard time understading this. Can you tell me what's the default value of gamma ,if for example, the input is a vector of 3 dimensions(3,) e.g. [3,3,3] and the number of input vectors are 10.000? Also, is there a way i can print it out to see its value?

Answer: This is easy to see with an example. The array X below has two features (columns). The variance of the array is 1.75. The default gamma is therefore is 1/(2*1.75) = 0.2857. You can verify this by checking the ._gamma attribute of the classifier. import numpy as np from sklearn.svm import SVC X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]]) y = np.array([1, 1, 2, 2]) clf = SVC(gamma='scale') clf.fit(X, y) n_features = X.shape[1] gamma = 1 / (n_features * X.var()) clf._gamma  Output: X Out[24]: array([[-1, -1], [-2, -1], [ 1, 1], [ 2, 1]]) n_features Out[25]: 2 X.var() Out[26]: 1.75 gamma Out[27]: 0.2857142857142857 clf._gamma Out[28]: 0.2857142857142857
Question: I have a bunch of modules. The modules and their imports are listed below: ast.py: import enum from abc import ABC, abstractmethod err.py: none lexer.py: from token import TokenTag, Token parser.py: from ast import * from err import UndeclaredIdentError, SyntaxError from token import TokenTag as Tag from type import Type peep.py: from lexer import Lexer from parser import Parser token.py: import enum treewalker.py: from abc import ABC, abstractmethod type.py: import enum from treewalker import TreeWalker  I tried to run peep.py but I get the following error: Traceback (most recent call last): File "peep.py", line 2, in <module> from parser import Parser ImportError: cannot import name 'Parser'  I don't understand why I got ImportError, I can't find any obvious circular dependencies in the file hierachy above. I did some research, I figured that I should rename the module ast.py to syntaxtree.py because ast.py already exists in Python's standard library. After renaming, it produced the same result. Any form of help is appreciated, thanks!

Answer: Open parser.py file and change the code for from parser import Parser to from .parser import Parser
Question: Essentially , what I want to do is , press a button that I program in Flutter and when that button is pressed , a python script should start running on my Android device .  I want to use youtube-dl (used to download Youtube videos) library in python but I wanna know if there is a way to run the library in flutter .  Any Help is appreciated . Thanks in advance . 

Answer: I know it's quite old now but may be someone else can get help out of it. There is library called starflut and it can incorporate python code inside your flutter app. Take a look here  https://pub.dev/packages/starflut
Question: I have two pandas data frames, a and b: a1 a2 a3 a4 a5 a6 a7 1 3 4 5 3 4 5 0 2 0 3 0 2 1 2 5 6 5 2 1 2  and b1 b2 b3 b4 b5 b6 b7 3 5 4 5 1 4 3 0 1 2 3 0 0 2 2 2 1 5 2 6 5  The two data frames contain exactly the same data, but in a different order and with different column names. Based on the numbers in the two data frames, I would like to be able to match each column name in a to each column name in b. It is not as easy as simply comparing the first row of a with the first row of b as there are duplicated values, for example both a4 and a7 have the value 5 so it is not possible to immediately match them to either b2 or b4. What is the best way to do this?

Answer: Here's one way leveraging broadcasting to check for equality between both dataframes and taking all on the result to check where all rows match. Then we can obtain indexing arrays for both dataframe's column names from the result of np.where (with @piR's contribution): i, j = np.where((a.values[:,None] == b.values[:,:,None]).all(axis=0)) dict(zip(a.columns[j], b.columns[i])) # {'a7': 'b2', 'a6': 'b3', 'a4': 'b4', 'a2': 'b7'}
Question: I'm trying to change the learning rate of my model after it has been trained with a different learning rate. I read here, here, here and some other places i can't even find anymore. I tried: model.optimizer.learning_rate.set_value(0.1) model.optimizer.lr = 0.1 model.optimizer.learning_rate = 0.1 K.set_value(model.optimizer.learning_rate, 0.1) K.set_value(model.optimizer.lr, 0.1) model.optimizer.lr.assign(0.1)  ... but none of them worked! I don't understand how there could be such confusion around such a simple thing. Am I missing something? EDIT: Working example Here is a working example of what I'd like to do: from keras.models import Sequential from keras.layers import Dense import keras import numpy as np model = Sequential() model.add(Dense(1, input_shape=(10,))) optimizer = keras.optimizers.Adam(lr=0.01) model.compile(loss='mse', optimizer=optimizer) model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50) # Change learning rate to 0.001 and train for 50 more epochs model.fit(np.random.randn(50,10), np.random.randn(50), initial_epoch=50, epochs=50) 

Answer: You can change the learning rate as follows: from keras import backend as K K.set_value(model.optimizer.learning_rate, 0.001)  Included into your complete example it looks as follows: from keras.models import Sequential from keras.layers import Dense from keras import backend as K import keras import numpy as np model = Sequential() model.add(Dense(1, input_shape=(10,))) optimizer = keras.optimizers.Adam(lr=0.01) model.compile(loss='mse', optimizer=optimizer) print("Learning rate before first fit:", model.optimizer.learning_rate.numpy()) model.fit(np.random.randn(50,10), np.random.randn(50), epochs=50, verbose=0) # Change learning rate to 0.001 and train for 50 more epochs K.set_value(model.optimizer.learning_rate, 0.001) print("Learning rate before second fit:", model.optimizer.learning_rate.numpy()) model.fit(np.random.randn(50,10), np.random.randn(50), initial_epoch=50, epochs=50, verbose=0)  I've just tested this with keras 2.3.1. Not sure why the approach didn't seem to work for you.
Question: I need to calculate period medians per seller ID (see simplyfied model below). The problem is I am unable to construct the ORM query. Model class MyModel: period = models.IntegerField(null=True, default=None) seller_ids = ArrayField(models.IntegerField(), default=list) aux = JSONField(default=dict)  Query queryset = ( MyModel.objects.filter(period=25) .annotate(seller_id=Func(F("seller_ids"), function="unnest")) .values("seller_id") .annotate( duration=Cast(KeyTextTransform("duration", "aux"), IntegerField()), median=Func( F("duration"), function="percentile_cont", template="%(function)s(0.5) WITHIN GROUP (ORDER BY %(expressions)s)", ), ) .values("median", "seller_id") )  ArrayField aggregation (seller_id) source  I think what I need to do is something along the lines below select t.*, p_25, p_75 from t join (select district, percentile_cont(0.25) within group (order by sales) as p_25, percentile_cont(0.75) within group (order by sales) as p_75 from t group by district ) td on t.district = td.district  above example source  Python 3.7.5, Django 2.2.8, Postgres 11.1

Answer: You can create a Median child class of the Aggregate class as was done by Ryan Murphy (https://gist.github.com/rdmurphy/3f73c7b1826cacee34f6c2a855b12e2e). Median then works just like Avg:  from django.db.models import Aggregate, FloatField class Median(Aggregate): function = 'PERCENTILE_CONT' name = 'median' output_field = FloatField() template = '%(function)s(0.5) WITHIN GROUP (ORDER BY %(expressions)s)'  Then to find the median of a field use  my_model_aggregate = MyModel.objects.all().aggregate(Median('period'))  which is then available as my_model_aggregate['period__median'].
Question: I would like to develop web apps in Google colab. The only issue is that you need a browser connected to local host to view the web app, but Google colab doesn't have a browser inside the notebook.  But it seems that there are ways around this. For example run_with_ngrok is a library for running flaks apps in colab/jupyter notebooks  https://github.com/gstaff/flask-ngrok#inside-jupyter--colab-notebooks When you use it, it gives a random address , "Running on http://.ngrok.io" And somehow the webapp that's running on Google colab is running on that address.  This is a great solution for Flask apps, but I am looking to run webapps in general on Google Colab, not just Flask ones. Is there a general method for running webapps in colab/jupyter notebooks?

Answer: You can plan to start a server on a port, e.g. port=8000. Find the URL to use this way. from google.colab.output import eval_js print(eval_js("google.colab.kernel.proxyPort(8000)")) # https://z4spb7cvssd-496ff2e9c6d22116-8000-colab.googleusercontent.com/  Then, start the server, e.g. !python -m http.server 8000  And click the first link above (instead of localhost or 127.0.0.1), it will open in a new tab. Display in cell You can display the result in an iframe in the output part. I made it into an easy function to call. from IPython.display import Javascript def show_port(port, height=400): display(Javascript(""" (async ()=>{ fm = document.createElement('iframe') fm.src = await google.colab.kernel.proxyPort(%s) fm.width = '95%%' fm.height = '%d' fm.frameBorder = 0 document.body.append(fm) })(); """ % (port, height) ))  Now you can start a webapp (here it is http.server) in a background. And display the result as an iframe below it. get_ipython().system_raw('python3 -m http.server 8888 &') show_port(8888)  To stop the server, you can call ps and kill the process.
Question: In many Docker tutorials based on Python (such as: this one) they use the option PYTHONDONTWRITEBYTECODE in order to make Python avoid to write .pyc files on the import of source modules (This is equivalent to specifying the -B option). What are the risks and advantages of setting this option up?

Answer: When you run a single python process in the container, which does not spawn other python processes itself during its lifetime, then there is no "risk" in doing that. Storing byte code on disk is used to compile python into byte code just upon the first invocation of a program and its dependent libraries to save that step upon the following invocations. In a container the process runs just once, therefore setting this option makes sense.
Question: My UploadForm class:  from app import app from flask_wtf.file import FileRequired, FileAllowed from wtforms.fields import MultipleFileField from wtforms.validators import InputRequired,DataRequired class UploadForm(FlaskForm): . . . roomImage = MultipleFileField('Room',validators=[FileAllowed(['jpg', 'png'], 'Image only!'), FileRequired('File was empty!')] ) . . .#there are other fields here which are not relevant to the problem at hand  HTML Template {% extends "base.html" %} {% block content %} <h1>Upload Your Images</h1> <form action="/" enctype="multipart/form-data" method="post" > {{ form.csrf_token }} Room<br /> {{form.roomImage()}} . . . <MORE THINGS THAT I HAVE EDITED OUT> {{form.submit()}} <br/> {% if form.errors %} {{ form.errors }} {% endif %} </form> {% endblock %}  hosts.py to run the check for validation def upload_image():#NEEDS HEAVY FIXING """home page to return the web page to upload documents""" form = UploadForm() if form.validate_on_submit():  Using VS's debugging tools, I find that form.validate_on_submit() doesn't work and always fails validation and I get this error on my html page.  {'roomImage': ['File was empty!']}  There is another MultipleFileField control with almost the exact same code. This issue does not happen when I use FileField to upload one file. The documentation on this is very limited and all I had to go on was this. I don't really know how to solve this issue. I have searched extensively for finding example involving MultipleFileField but they are not using any validation. A thread on Github that I can't find anymore suggested using OptionalValidator, but then that is not an option for me and even that didn't work. Can someone suggest me a solution? EDIT: Even the FileAllowed() validator does not seem to work.

Answer: This works for me (found on GitHub "between the lines"): multi_file = MultipleFileField("Upload File(s)", validators=[DataRequired()])  However  FileAllowed(["xml", "jpg"])  is ignored and does not work for me. EDIT: No, sadly, it does not work... It returns True for form.validate() and for form.validate_on_submit() but when you pass no files, by deleting required=""  from <input id="multi_file" multiple="" name="multi_file" required="" type="file">  and submit a form, it still evaluate that as True. So problem sill persist in full, as described...
Question: I find violin plots very informative and useful, I use python library 'seaborn'. However, when applied to positive values, they nearly always show negative values at the lower end. I find this really misleading, especially when working with real-life datasets. In the official documentation of seaborn https://seaborn.pydata.org/generated/seaborn.violinplot.html one can see examples with "total_bill" and "tip" which can not be negative. The violin plots show negative values, however. For example, import seaborn as sns sns.set(style="whitegrid") tips = sns.load_dataset("tips") ax = sns.violinplot(x="day", y="total_bill", hue="smoker",data=tips, palette="muted", split=True)   I do understand, that those negative values come from gaussian kernels. My question is, therefore: is there any way to solve this problem? Another library in python? Possibility to specify a different kernel? 

Answer: You can use the keyword cut=0 to limit your plot to the data range. If the data doesn't have negative values, this will chop the end of the violin to zero. Using the same example as you, try:  ax = sns.violinplot(x="day", y="total_bill", hue="smoker",data=tips, palette="muted", split=True,cut=0)
Question: I've recently downloaded all packages from PyPI. One interesting observation was that of the Top-15 of the biggest packages, all execept one are deep learning packages:  mxnet: mxnet-cu90 (600 MB), mxnet-cu92, mxnet-cu101mkl, mxnet-cu101 (and 6 more mxnet versions) cntk: cntk-gpu (493MB) H2O4GPU (366MB) tensorflow: tensorflow-gpu (357MB), tensorflow  I looked at mxnet-cu90. It has exactly one huge file: libmxnet.so (936.7MB). What does this file contain? Is there any way to make it smaller? I'm especially astonished that those libraries are so huge considering that one usually uses them on top of CUDA + cuDNN, which I thought would do the heavy lifting. As a comparison, I looked at related libraries with which you can also build deep learning libraries:  numpy: 6MB sympy: 6MB pycuda: 3.6MB tensorflow-cpu: 116MB (so the GPU version needs 241 MB more or around 3x the size!)  

Answer: Deep learning frameworks are large because they package CuDNN from NVIDIA into their wheels. This is done for the convenience of downstream users.  CuDNN are the primitives that the frameworks call to execute highly optimised neural network ops (e.g. LSTM) The unzipped version of CuDNN for windows 10 is 435MB.
Question: While importing, Python (Anaconda) gives the following error: ImportError: cannot import name 'PILLOW_VERSION' from 'PIL'  I tried removing pillow and then conda install but the error persists.

Answer: Pillow 7.0.0 removed PILLOW_VERSION, you should use __version__ in your own code instead.  https://pillow.readthedocs.io/en/stable/deprecations.html#pillow-version-constant   Edit (2020-01-16):  If using torchvision, this has been fixed in v0.5.0. To fix:  Require torchvision>=0.5.0 If Pillow was temporarily pinned, remove the pin    Old info (2020-01-09): If using torchvision, there is a release planned this week (week 2, 2020) to fix it:  https://github.com/pytorch/vision/issues/1712#issuecomment-570286349  The options are:  wait for the new torchvision release use the master version of torchvision (eg. pip install -U git+https://github.com/pytorch/vision) install torchvision from a nightly, which also requires a pytorch from a nightly version or install Pillow<7 (eg. pip install "pillow<7")
Question: I am trying to implement a DQL model on one game of openAI gym. But it's giving me following error.  TypeError: len is not well defined for symbolic Tensors. (activation_3/Identity:0) Please call x.shape rather than len(x) for shape information.  Creating a gym environment: ENV_NAME = 'CartPole-v0' env = gym.make(ENV_NAME) np.random.seed(123) env.seed(123) nb_actions = env.action_space.n  My model looks like this: model = Sequential() model.add(Flatten(input_shape=(1,) + env.observation_space.shape)) model.add(Dense(16)) model.add(Activation('relu')) model.add(Dense(nb_actions)) model.add(Activation('linear')) print(model.summary())  Fitting that model to DQN model from keral-rl as follows: policy = EpsGreedyQPolicy() memory = SequentialMemory(limit=50000, window_length=1) dqn = DQNAgent(model=model, nb_actions=nb_actions, memory=memory, nb_steps_warmup=10, target_model_update=0.001, policy=policy) dqn.compile(Adam(lr=1e-3), metrics=['mse', 'mae']) dqn.fit(env, nb_steps=5000, visualize=False, verbose=3)  The error is from this line: dqn = DQNAgent(model=model, nb_actions=nb_actions, memory=memory, nb_steps_warmup=10, target_model_update=0.001, policy=policy)  I am using keras-rl==0.4.2 and tensorflow==2.1.0. Based on other answers, I also tried tensorflow==2.0.0-beta0 but it doesn't solve the error.  Can someone please explain to me why I am facing this error? and how to solve it? Thank you.

Answer: The reason this breaks is because, tf.Tensor TF 2.0.0 (and TF 1.15) has the __len__ overloaded and raises an exception. But TF 1.14 for example doesn't have the __len__ attribute. Therefore, anything TF 1.15+ (inclusive) breaks keras-rl (specifically here), which gives you the above error. So you got two options,  Downgrade to TF 1.14 (recommended) Delete the __len__ overloading in TensorFlow source (not recommended as this can break other things)
Question: I have a dataframe (df_full) like so: |cust_id|address |store_id|email |sales_channel|category| ------------------------------------------------------------------- |1234567|123 Main St|10SjtT |idk@gmail.com|ecom |direct | |4567345|345 Main St|10SjtT |101@gmail.com|instore |direct | |1569457|876 Main St|51FstT |404@gmail.com|ecom |direct |  and I would like to combine the last 4 fields into one metadata field that is a dictionary like so: |cust_id|address |metadata | ------------------------------------------------------------------------------------------------------------------- |1234567|123 Main St|{'store_id':'10SjtT', 'email':'idk@gmail.com','sales_channel':'ecom', 'category':'direct'} | |4567345|345 Main St|{'store_id':'10SjtT', 'email':'101@gmail.com','sales_channel':'instore', 'category':'direct'}| |1569457|876 Main St|{'store_id':'51FstT', 'email':'404@gmail.com','sales_channel':'ecom', 'category':'direct'} |  is that possible? I've seen a few solutions around on stack overflow but none of them address combining more than 2 fields into a dictionary field.

Answer: Use to_dict, columns = ['store_id', 'email', 'sales_channel', 'category'] df['metadata'] = df[columns].to_dict(orient='records')  And if you want to drop original columns, df = df.drop(columns=columns)
Question: I am trying to predict car prices (by machine learning) with a simple linear regression (only one independent variable). The variables are "highway miles per gallon" 0 27 1 27 2 26 3 30 4 22 .. 200 28 201 25 202 23 203 27 204 25 Name: highway-mpg, Length: 205, dtype: int64  and "price": 0 13495.0 1 16500.0 2 16500.0 3 13950.0 4 17450.0 ... 200 16845.0 201 19045.0 202 21485.0 203 22470.0 204 22625.0 Name: price, Length: 205, dtype: float64  With the following code: from sklearn.linear_model import LinearRegression x = df["highway-mpg"] y = df["price"] lm = LinearRegression() lm.fit([x],[y]) Yhat = lm.predict([x]) print(Yhat) print(lm.intercept_) print(lm.coef_)  However, the intercept and slope coefficient print commands give me the following output: [[0. 0. 0. ... 0. 0. 0.] [0. 0. 0. ... 0. 0. 0.] [0. 0. 0. ... 0. 0. 0.] ... [0. 0. 0. ... 0. 0. 0.] [0. 0. 0. ... 0. 0. 0.] [0. 0. 0. ... 0. 0. 0.]]  Why doesn't it print the intercept and slope coefficient? The "Yhat" print command does print out the predicted values in an array properly, but somehow the other print commands do not print my desired output...

Answer: Essentially, what caused the strange looking coef_ and intercept_ was the fact that your data had 205 features and 205 targets with only 1 sample. Definitely not what you wanted! You probably want 1 feature, 205 samples, and 1 target. To do this, you need to reshape your data: from sklearn.linear_model import LinearRegression import numpy as np mpg = np.array([27, 27, 26, 30, 22, 28, 25, 23, 27, 25]).reshape(-1, 1) price = np.array([13495.0, 16500.0, 16500.0, 13950.0, 17450.0, 16845.0, 19045.0, 21485.0, 22470.0, 22625.0]) lm = LinearRegression() lm.fit(mpg, price) print(lm.intercept_) print(lm.coef_)  I used the arrays there for testing, but obviously use the data from your dataframe. P.S. If you omit the resize, you get an error message like this: ValueError: Expected 2D array, got 1D array instead: array=[27 27 26 30 22 28 25 23 27 25]. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.  ^ It tells you what to do!