eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Whenever i execute the code below it gives me following Error: ImportError: cannot import name 'Serial' from 'serial' (unknown location)  Code:  from serial import Serial arduinodata = Serial('com4',9600) print("Enter n to ON LED and f to OFF LED") while 1: input_data = raw_input() print ("You Entered"+input_data) if (input_data == 'n'): arduinodata.write(b'1') print("LED ON") if (input_data == 'f'): arduinodata.write(b'0') print("LED OFF")  I have installed all the required python modules. Like pyserial, pyfirmata etc but it is continuously giving me this error.

Answer: Most likely missing an  __init__.py  file or the module, or the file sub-directory for the module (Serial) is on a different layer than the file executable. Hope that helps :).
Question: I think that Adam optimizer is designed such that it automtically adjusts the learning rate. But there is an option to explicitly mention the decay in the Adam parameter options in Keras. I want to clarify the effect of decay on Adam optimizer in Keras. If we compile the model using decay say 0.01 on lr = 0.001, and then fit the model running for 50 epochs, then does the learning rate get reduced by a factor of 0.01 after each epoch? Is there any way where we can specify that the learning rate should decay only after running for certain number of epochs? In pytorch there is a different implementation called AdamW, which is not present in the standard keras library. Is this the same as varying the decay after every epoch as mentioned above? Thanks in advance for the reply.

Answer: From source code, decay adjusts lr per iterations according to lr = lr * (1. / (1. + decay * iterations)) # simplified  see image below. This is epoch-independent. iterations is incremented by 1 on each batch fit (e.g. each time train_on_batch is called, or how many ever batches are in x for model.fit(x) - usually len(x) // batch_size batches). To implement what you've described, you can use a callback as below: from keras.callbacks import LearningRateScheduler def decay_schedule(epoch, lr): # decay by 0.1 every 5 epochs; use `% 1` to decay after each epoch if (epoch % 5 == 0) and (epoch != 0): lr = lr * 0.1 return lr lr_scheduler = LearningRateScheduler(decay_schedule) model.fit(x, y, epochs=50, callbacks=[lr_scheduler])  The LearningRateScheduler takes a function as an argument, and the function is fed the epoch index and lr at the beginning of each epoch by .fit. It then updates lr according to that function - so on next epoch, the function is fed the updated lr. Also, there is a Keras implementation of AdamW, NadamW, and SGDW, by me - Keras AdamW.    Clarification: the very first call to .fit() invokes on_epoch_begin with epoch = 0 - if we don't wish lr to be decayed immediately, we should add a epoch != 0 check in decay_schedule. Then, epoch denotes how many epochs have already passed - so when epoch = 5, the decay is applied.
Question: I am trying to plot image data in altair, specifically trying to replicate face recognition example in this link from Jake VDP's book - https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html. Any one had luck plotting image data in altair?

Answer: Altair features an image mark that can be used if you want to plot images that are available at a URL; for example: import altair as alt import pandas as pd source = pd.DataFrame.from_records([ {"x": 0.5, "y": 0.5, "img": "https://vega.github.io/vega-datasets/data/ffox.png"}, {"x": 1.5, "y": 1.5, "img": "https://vega.github.io/vega-datasets/data/gimp.png"}, {"x": 2.5, "y": 2.5, "img": "https://vega.github.io/vega-datasets/data/7zip.png"} ]) alt.Chart(source).mark_image( width=50, height=50 ).encode( x='x', y='y', url='img' )   Altair is not as well suited to displaying 2-dimensional data arrays as images, because the grammar is primarily designed to work with structured tabular data. However, it is possible to do using a combination of flatten transforms and window transforms. Here is an example using the data from the page you linked to: import altair as alt import pandas as pd from sklearn.datasets import fetch_lfw_people faces = fetch_lfw_people(min_faces_per_person=60) data = pd.DataFrame({ 'image': list(faces.images[:12]) # list of 2D arrays }) alt.Chart(data).transform_window( index='count()' # number each of the images ).transform_flatten( ['image'] # extract rows from each image ).transform_window( row='count()', # number the rows... groupby=['index'] # ...within each image ).transform_flatten( ['image'] # extract the values from each row ).transform_window( column='count()', # number the columns... groupby=['index', 'row'] # ...within each row & image ).mark_rect().encode( alt.X('column:O', axis=None), alt.Y('row:O', axis=None), alt.Color('image:Q', scale=alt.Scale(scheme=alt.SchemeParams('greys', extent=[1, 0])), legend=None ), alt.Facet('index:N', columns=4) ).properties( width=100, height=120 )
Question: I came from Java where we can avoid calling super class zero-argument constructor. The call to it is generated implicitly by the compiler.  I read this post about super() and now in question about is it really necessary to do something like this explicitly: class A(object): def __init__(self): print("world") class B(A): def __init__(self): print("hello") super().__init__() #Do we get some Undefined Behavior if we do not call it explicitly? 

Answer: If you override the __init__ method of the superclass, then the __init__ method of the subclass needs to explicitly call it if that is the intended behavior, yes. Your mental model of __init__ is incorrect; it is not the constructor method, it is a hook which the constructor method calls to let you customize object initialization easily. (The actual constructor is called __new__ but you don't need to know this, and will probably never need to interact with it directly, let alone change it.)
Question: Using Pandas 1.0.0, how can I change the time of a datetime dataframe column to midnight in one line of code? e.g.: from START_DATETIME 2017-02-13 09:13:33 2017-03-11 23:11:35 2017-03-12 00:44:32 ...  to START_DATETIME 2017-02-13 00:00:00 2017-03-11 00:00:00 2017-03-12 00:00:00 ...  My attempt: df['START_DATETIME'] = df['START_DATETIME'].apply(lambda x: pd.Timestamp(x).replace(hour=0, minute=0, second=0))  but this produces START_DATETIME 2017-02-13 2017-03-11 2017-03-12 ... 

Answer: Your method already converted datetime values correctly to midnight. I.e., their time are 00:00:00. Pandas just intelligently doesn't show the time part because it is redundant to show all same time of 00:00:00. After you assigning result back to START_DATETIME, print a cell will show  print(df.loc[0, START_DATETIME]) Output: 2017-02-13 00:00:00  Besides, to convert time to 00:00:00, you should use dt.normalize or dt.floor df['START_DATETIME'] = pd.to_datetime(df['START_DATETIME']).dt.normalize()  or  df['START_DATETIME'] = pd.to_datetime(df['START_DATETIME']).dt.floor('D')   If you want to force pandas to show 00:00:00 in the series output, you need convert START_DATETIME to str after converting pd.to_datetime(df['START_DATETIME']).dt.floor('D').dt.strftime('%Y-%m-%d %H:%M:%S') Out[513]: 0 2017-02-13 00:00:00 1 2017-03-11 00:00:00 2 2017-03-12 00:00:00 Name: START_DATETIME, dtype: object
Question: I am using Google Colab python 3.x and I have a Dataframe as below. I would like to see all cells on each row and column. How can I do this? I tried pd.set_option('display.max_columns', 3000) but it didn't work. # importing pandas as pd import pandas as pd # dictionary of lists dict = {'name':["a1", "b2", "c2", "d3"], 'degree': ["We explained to customer how correct fees (100) were charged. Account balance was too low", "customer was late in paying fees and we have to charge fine", "customer's credit score was too low and we have to charge higher interest rate", "customer complained a lot and didnt listen to our explanation. I had to escalate the call"], 'score':[90, 40, 80, 98]} # creating a dataframe from a dictionary df = pd.DataFrame(dict) print (df) name degree score 0 a1 We explained to customer how correct fees (100... 90 1 b2 customer was late in paying fees and we have t... 40 2 c2 customer's credit score was too low and we hav... 80 3 d3 customer complained a lot and didnt listen to ... 98 

Answer: use pd.set_option('max_colwidth', <width>) for column width & pd.set_option('max_rows', <rows>) for number of rows. see https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html [] pd.set_option('max_rows', 99999) [] pd.set_option('max_colwidth', 400) [] pd.describe_option('max_colwidth') display.max_colwidth : int The maximum width in characters of a column in the repr of a pandas data structure. When the column overflows, a "..." placeholder is embedded in the output. [default: 50] [currently: 400] [] df = pd.DataFrame(d) [] df name degree score 0 a1 We explained to customer how correct fees (100) were charged. Account balance was too low 90 1 b2 customer was late in paying fees and we have to charge fine 40 2 c2 customer's credit score was too low and we have to charge higher interest rate 80 3 d3 customer complained a lot and didnt listen to our explanation. I had to escalate the call 98
Question: I want to randomly choose 2 elements out of a list. >>> import random >>> random.sample(["foo", "bar", "baz", "quux"], 2) ['quux', 'bar']  But I want to use a numpy.random.Generator to do it, rather than using Python's global random number generator. Is there a built-in or easy way to do this? >>> import numpy as np >>> gen = np.random.default_rng() >>> ???  [edit] the point is to make use of gen which allows you to seed it for reproducibility. I realize the same can hypothetically be accomplished by re-seeding global generators, but I specifically want to use gen, a local generator, rather than relying on global generators.

Answer: If you really want to do it from the numpy.random.Generator: import numpy as np gen = np.random.default_rng() gen.choice(["foo", "bar", "baz", "quux"], 2, replace=False)  Note that np.random.choice selects with replacement by default (i.e. each item can be sampled multiple times), so turn this off if you want an equivalent method to random.sample (credit: @ayhan).
Question: I'm trying to calculate the Levenshtein distance between two Pandas columns but I'm getting stuck Here is the library I'm using. Here is a minimal, reproducible example: import pandas as pd from textdistance import levenshtein attempts = [['passw0rd', 'pasw0rd'], ['passwrd', 'psword'], ['psw0rd', 'passwor']] df=pd.DataFrame(attempts, columns=['password', 'attempt'])   password attempt 0 passw0rd pasw0rd 1 passwrd psword 2 psw0rd passwor  My poor attempt: df.apply(lambda x: levenshtein.distance(*zip(x['password'] + x['attempt'])), axis=1)  This is how the function works. It takes two strings as arguments: levenshtein.distance('helloworld', 'heloworl')  Out[1]: 2 

Answer: Maybe I'm missing something, is there a reason you don't like the lambda expression? This works to me: import pandas as pd from textdistance import levenshtein attempts = [['passw0rd', 'pasw0rd'], ['passwrd', 'psword'], ['psw0rd', 'passwor'], ['helloworld', 'heloworl']] df=pd.DataFrame(attempts, columns=['password', 'attempt']) df.apply(lambda x: levenshtein.distance(x['password'], x['attempt']), axis=1)  out: 0 1 1 3 2 4 3 2 dtype: int64
Question: I'm looking for a way to get a list of all installed/importable python modules from a within a Jupyterlab notebook. From the command line, I can get the list by running py -3 -m pip freeze  (or) pip freeze  In the Jupyterlab console, running pip freeze returns The following command must be run outside of the IPython shell: $ pip freeze The Python package manager (pip) can only be used from outside of IPython. Please reissue the `pip` command in a separate terminal or command prompt. See the Python documentation for more information on how to install packages: https://docs.python.org/3/installing/  For older versions of pip, it was possible to import pip and get a list from within a notebook. The command was  help('modules')  This now gives a warning and returns nothing. c:\python37\lib\site-packages\IPython\kernel\__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead. "You should import from ipykernel or jupyter_client instead.", ShimWarning)  10 year old stackoverflow solutions like How can I get a list of locally installed Python modules? also no longer work. Is there a proper way of doing this (without using a subprocess hack or running pip as an external program like "!pip")

Answer: import pip._internal.operations.freeze _ = pip._internal.operations.freeze.get_installed_distributions() print(sorted(["%s==%s" % (i.key, i.version) for i in _])[:10]) ['absl-py==0.7.1', 'aiml==0.9.2', 'aio-utils==0.0.1', 'aiocache==0.10.1', 'aiocontextvars==0.2.2', 'aiocqhttp==0.6.7', 'aiodns==2.0.0', 'aiofiles==0.4.0', 'aiohttp-proxy==0.1.1', 'aiohttp==3.6.2']  This works in Win10 with Python 3.6 & 3.7 (ipython, pip.version: '20.0.1') at least. I took a look at the source code in Lib\site-packages\pip.
Question: In official documents of tensorflow.keras,   validation_data could be: tuple (x_val, y_val) of Numpy arrays or tensors tuple (x_val, y_val, val_sample_weights) of Numpy arrays dataset For the first two cases, batch_size must be provided. For the last case, validation_steps could be provided.  It does not mention if generator could act as validation_data. So I want to know if validation_data could be a datagenerator? like the following codes: net.fit_generator(train_it.generator(), epoch_iterations * batch_size, nb_epoch=nb_epoch, verbose=1, validation_data=val_it.generator(), nb_val_samples=3, callbacks=[checker, tb, stopper, saver])  Update: In the official documents of keras, the same contents, but another sentense is added:   dataset or a dataset iterator   Considering that   dataset For the first two cases, batch_size must be provided. For the last case, validation_steps could be provided.  I think there should be 3 cases. Keras' documents are correct. So I will post an issue in tensorflow.keras to update the documents.

Answer: Yes it can, that's strange that it is not in the doc but is it working exactly like the x argument, you can also use a keras.Sequence or a generator. In my project I often use keras.Sequence that acts like a generator  Minimum working example that shows that it works : import numpy as np from tensorflow.keras import Sequential from tensorflow.keras.layers import Dense, Flatten def generator(batch_size): # Create empty arrays to contain batch of features and labels batch_features = np.zeros((batch_size, 1000)) batch_labels = np.zeros((batch_size,1)) while True: for i in range(batch_size): yield batch_features, batch_labels model = Sequential() model.add(Dense(125, input_shape=(1000,), activation='relu')) model.add(Dense(8, activation='relu')) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) train_generator = generator(64) validation_generator = generator(64) model.fit(train_generator, validation_data=validation_generator, validation_steps=100, epochs=100, steps_per_epoch=100)   100/100 [==============================] - 1s 13ms/step - loss: 0.6689 - accuracy: 1.0000 - val_loss: 0.6448 - val_accuracy: 1.0000 Epoch 2/100 100/100 [==============================] - 0s 4ms/step - loss: 0.6223 - accuracy: 1.0000 - val_loss: 0.6000 - val_accuracy: 1.0000 Epoch 3/100 100/100 [==============================] - 0s 4ms/step - loss: 0.5792 - accuracy: 1.0000 - val_loss: 0.5586 - val_accuracy: 1.0000 Epoch 4/100 100/100 [==============================] - 0s 4ms/step - loss: 0.5393 - accuracy: 1.0000 - val_loss: 0.5203 - val_accuracy: 1.0000
Question: I have some classes that inherit from each other. All classes contain the same method (let us call it mymethod), whereby the children overwrite the base class method. I want to generate a documentation for mymethod in all classes using sphinx.  Suppose mymethod takes an argument myargument. This argument has the same type and meaning for both the base method as well as the inherited method. To minimize redundancies, I would like to write the documentation for myargument only for the base class and insert the documentation in the child method's documentation. That is, I do not want to only put a simple reference to the base class but rather dynamically insert the text when I generate the documentation. Can this be done? How? Below please find some code illustrating the problem. class BaseClass def mymethod(myargument): """This does something Params ------ myargument : int Description of the argument """ [...] class MyClass1(BaseClass): def mymethod(myargument): """This does something Params ------ [here I would like to insert in the description of ``myargument`` from ``BaseClass.mymethod``] """ BaseClass.mymethod(myargument) [...] class MyClass2(BaseClass): def mymethod(myargument, argument2): """This does something Params ------ [here I would like to insert in the description of ``myargument`` in ``BaseClass.mymethod``] argument2 : int Description of the additional argument """ BaseClass.mymethod(argument) [...] 

Answer: Probably not ideal, but maybe you could use a decorator to extend the docstring. For example: class extend_docstring: def __init__(self, method): self.doc = method.__doc__ def __call__(self, function): if self.doc is not None: doc = function.__doc__ function.__doc__ = self.doc if doc is not None: function.__doc__ += doc return function class BaseClass: def mymethod(myargument): """This does something Params ------ myargument : int Description of the argument """ [...] class MyClass1(BaseClass): @extend_docstring(BaseClass.mymethod) def mymethod(myargument): BaseClass.mymethod(myargument) [...] class MyClass2(BaseClass): @extend_docstring(MyClass1.mymethod) def mymethod(myargument, argument2): """argument2 : int Description of the additional argument """ BaseClass.mymethod(argument) [...] print('---BaseClass.mymethod---') print(BaseClass.mymethod.__doc__) print('---MyClass1.mymethod---') print(MyClass1.mymethod.__doc__) print('---MyClass2.mymethod---') print(MyClass2.mymethod.__doc__)  Result: ---BaseClass.mymethod--- This does something Params ------ myargument : int Description of the argument ---MyClass1.mymethod--- This does something Params ------ myargument : int Description of the argument ---MyClass2.mymethod--- This does something Params ------ myargument : int Description of the argument argument2 : int Description of the additional argument  The override method could be resolved dynamically if you make the decorator a descriptor and search for it into __get__ but that means the decorator is no longer stackable as it doesn't return the real function.
Question: I'm trying to detect all the rectangles from the relational database. But some of the boxes are not being detected by my script. Please help me to do that. Thank you. The Image:  My Code: #!/usr/bin/python import cv2 import numpy as np im = cv2.imread("table.png") image = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY) thresh = cv2.threshold(image,0,255,cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1] edge = cv2.Canny(thresh,30,200) cont = cv2.findContours(edge,cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)[0] for j,i in enumerate(cont): x,y,w,h = cv2.boundingRect(i) if (w*h>900): cv2.drawContours(image,[i],0,(0,0,255),3) cv2.imshow("Image",image) cv2.waitKey(0)  OUTPUT: 

Answer: Here's an simple approach using thresholding + morphological operations.  Obtain binary image. Load image, convert to grayscale, then adaptive threshold Fill rectangular contours. Find contours and fill the contours to create filled rectangular blocks. Perform morph open. We create a rectangular structuring element and morph open to remove the lines Draw rectangle. Find contours and draw bounding rectangles.   Here's each step visualized:  Using this screenshotted image (contains more border since the provided image has the rectangles too close to the border). You could add a border to the input image instead of screenshotting for more border area. Take a look at add border to image  Binary image  Filled rectangular contours  Morph open  Result   Code import cv2 # Load iamge, grayscale, adaptive threshold image = cv2.imread('1.png') result = image.copy() gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) thresh = cv2.adaptiveThreshold(gray,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV,51,9) # Fill rectangular contours cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) cnts = cnts[0] if len(cnts) == 2 else cnts[1] for c in cnts: cv2.drawContours(thresh, [c], -1, (255,255,255), -1) # Morph open kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (9,9)) opening = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel, iterations=4) # Draw rectangles cnts = cv2.findContours(opening, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) cnts = cnts[0] if len(cnts) == 2 else cnts[1] for c in cnts: x,y,w,h = cv2.boundingRect(c) cv2.rectangle(image, (x, y), (x + w, y + h), (36,255,12), 3) cv2.imshow('thresh', thresh) cv2.imshow('opening', opening) cv2.imshow('image', image) cv2.waitKey()  Note: Depending on the image, you may have to modify the kernel size. For instance, it may be necessary to increase the kernel from (5, 5) to say (11, 11). In addition, you could increase or decrease the number of iterations when performing cv2.morphologyEx(). There is a trade-off when increasing or decreasing the kernel size as you may remove more or less of the lines. Again, it all varies depending on the input image.
Question: Problem Statment I'm trying to run some old python code that requires scikit-learn 18.0 but the current version I have installed is 0.22 and so I'm getting a warning/invalid data when I run the code.  What I've Tried I tried installing the specific version both in the terminal: python -m pip install scikit-learn==0.18 and in conda and none of that has worked. All I can install is v 0.22. Help? Thanks.  Error In Terminal  ERROR: Failed building wheel for scikit-learn Running setup.py clean for scikit-learn Failed to build scikit-learn Installing collected packages: scikit-learn Found existing installation: scikit-learn 0.22.1 Uninstalling scikit-learn-0.22.1: Successfully uninstalled scikit-learn-0.22.1 Running setup.py install for scikit-learn ... error ERROR: Command errored out with exit status 1:  Error through conda environment:  PackagesNotFoundError: The following packages are not available from current channels: - scikit-learn==0.18 this was after creating and activating the new environment 

Answer: Tackling your issues one at a time:  python -m pip install scikit-learn==0.18 fails  This is probably due to the fact that scikit-learn==0.18, if you check on pypi only has whl files for python 3.5 and 2.7 for windows, therefore pip downloads the source distribution and then fails in compiling it, probably because it doesn't work with newer python versions  The following packages are not available from current channels  This happens, because scikit-learn==18.0 simply does not exist in the default conda channels. On my win64 machine, the oldesst version that I can install is 19.0 (You can check by typing conda search scikit-learn in the cmd), so unfortunately no way to install using the default conda channels. There is a channel called free (found through the anaconda website) that has scikit-learn 18.0, so you could install with: conda install -c free scikit-learn  To also make sure that the python version is compatible, I would just create a fitting environment: conda create -n py35 -c free scikit-learn=0.18.0