eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a Jupyter Notebook and I would like to use some credentials I have put into a config.py file. This file is in the same folder as the Jupyter Notebook. I use the line import config  The problem is Jupyter replies with this message: ModuleNotFoundError: No module named 'config.py'; 'config' is not a package  Thank you for your help

Answer: After some research, I have found a way to solve my need using Dotenv Python Package: pypi.org/project/python-dotenv What needs to be done? Insert the following lines: !pip install python-dotenv # Credentials file %load_ext dotenv %dotenv import os  Then place a hidden file named .env where the credentials are places. In my case it looks like this: CLIENT_ID = "XXXX" CLIENT_SECRET = "YYYY"  Then, in your notebook file, whenever you need to load the credentials use: CLIENT_ID = os.getenv("CLIENT_ID") CLIENT_SECRET = os.getenv("CLIENT_SECRET")  And you are good to go! Credit to http://veekaybee.github.io/2020/02/25/secrets/ for the solution
Question: I have a basic CNN model's code built with tensorflow.keras library: model = Sequential() # First Layer model.add(Conv2D(64, (3,3), input_shape = (IMG_SIZE,IMG_SIZE,1))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size = (3,3))) # Second Layer model.add(Conv2D(64, (3,3))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size = (3,3))) # Third Layer model.add(Conv2D(64, (3,3))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size = (3,3))) # Fourth Layer model.add(Conv2D(64, (3,3))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size = (3,3))) # Fifth Layer model.add(Conv2D(64, (3,3))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size = (3,3))) model.add(Flatten()) # Sixth Layer model.add(Dense(64)) model.add(Activation("relu")) # Seventh Layer model.add(Dense(1)) model.add(Activation('sigmoid'))  Now, I want to make a connection between the second and the fourth layer to achieve a residual block using tensorflow.keras library. So, How should I modify the code to achieve such a residual block?

Answer: Residual Block from ResNet Architecture is the following :  You need to use the Keras functionnal API because Sequential models are too limited. Its implementation in Keras is : from tensorflow.keras import layers def resblock(x, kernelsize, filters): fx = layers.Conv2D(filters, kernelsize, activation='relu', padding='same')(x) fx = layers.BatchNormalization()(fx) fx = layers.Conv2D(filters, kernelsize, padding='same')(fx) out = layers.Add()([x,fx]) out = layers.ReLU()(out) out = layers.BatchNormalization()(out) return out  BatchNormalisation()layer is not essential but may be a solid option to increase its accuracy. x needs also to have the same number of filters than the filters parameter.
Question: I would like to add a semi-transparent rectangle filled with a solid colour to an already loaded semi-transparent PNG. Here's an example input image I am using:  That image is loaded with a standard cv2.IMREAD_UNCHANGED flag so that alpha channel is perfectly preserved. That input image is stored in the image variable. Here's my code that I have so far: # get image dimensions imgHeight, imgWidth = image.shape[:2] # create empty overlay layer with 4 channels overlay = np.zeros((imgHeight, imgWidth, 4), dtype = "uint8") # draw semi-transparent red rectangle overlay[200:300, 0:imgWidth] = (0, 0, 255, 200) # extract alpha channel from overlay alpha = cv2.split(overlay)[3] # compute mask mask = (np.multiply(alpha, 1.0 / 255))[:, :, np.newaxis] # blend input image and overlay output = cv2.convertScaleAbs(overlay * mask + image * (1 - mask))  And here's the result that I am getting:  At first glance it looks acceptable. We have our input image with a semi-transparent rectangle in the middle. However, upon closer inspection, we can observe strange behaviour when mixing alpha channels (marked with arrows):  It seems that alpha is not blended at all which results in original image pixels being only fully opaque or fully transparent. Perhaps my method of blending transparent PNG with semi-transparent shapes is far from ideal. As a side note, I did try the cv2.addWeighted method but that yielded even worse results. I would like the solution to be limited to OpenCV and/or Numpy. Any help would be greatly appreciated.

Answer: As unlut pointed out this is indeed a duplicate. Just in case someone stumbles on it, Mark Setchell's answer works pretty well: # get image dimensions imgHeight, imgWidth = image.shape[:2] # create empty overlay layer with 4 channels overlay = np.zeros((imgHeight, imgWidth, 4), dtype = "uint8") # draw semi-transparent red rectangle overlay[200:300, 0:imgWidth] = (0, 0, 255, 200) # Extract the RGB channels srcRGB = image[...,:3] dstRGB = overlay[...,:3] # Extract the alpha channels and normalise to range 0..1 srcA = image[...,3]/255.0 dstA = overlay[...,3]/255.0 # Work out resultant alpha channel outA = srcA + dstA*(1-srcA) # Work out resultant RGB outRGB = (srcRGB*srcA[...,np.newaxis] + dstRGB*dstA[...,np.newaxis]*(1-srcA[...,np.newaxis])) / outA[...,np.newaxis] # Merge RGB and alpha (scaled back up to 0..255) back into single image outRGBA = np.dstack((outRGB,outA*255)).astype(np.uint8) plt.imshow(outRGBA)
Question: I was wondering if numpy could be used to build the most basic cube model where all cross-combinations and their computed value are stored. Let's take the following example of data: AUTHOR BOOK YEAR SALES Shakespeare Hamlet 2000 104.2 Shakespeare Hamlet 2001 99.0 Shakespeare Romeo 2000 27.0 Shakespeare Romeo 2001 19.0 Dante Inferno 2000 11.6 Dante Inferno 2001 12.6  And to be able to build something like:  YEAR TOTAL AUTHOR BOOK 2000 2001 (ALL) (ALL) 142.8 130.6 273.4 Shakespeare (ALL) 131.2 118.0 249.2 Dante (ALL) 11.6 12.6 24.2 Shakespeare Hamlet 104.2 99.0 203.2 Shakespeare Romeo 27.0 19.0 46.0 Dante Inferno 11.6 12.6 24.2  I'm hoping that the usage of using something like meshgrid might get me 75% there. Basically, I'd like to see if it's possible to build a structure of all pre-computed values with numpy (not pandas) to build a structure so that I could retrieve the above result of all possible combination. For the sake of simplicity, let's only consider the SUM as the only possible calculation. Perhaps this is a roundable way of asking, but could numpy be the backbone of doing this, or do I need to use something else? And finally, if not possible in numpy how might this be stored in a MDA?

Answer: I think numpy record arrays can be used for this task, below is my solution based on record arrays. class rec_array(): def __init__(self,author=None,book=None,year=None,sales=None): self.dtype = [('author','<U20'), ('book','<U20'),('year','<U20'),('sales',float)] self.rec_array = np.rec.fromarrays((author,book,year,sales),dtype=self.dtype) def add_record(self,author,book,year,sales): new_rec = np.rec.fromarrays((author,book,year,sales),dtype=self.dtype) if not self.rec_array.shape == (): self.rec_array = np.hstack((self.rec_array,new_rec)) else: self.rec_array = new_rec def get_view(self,conditions): """ conditions: A list of conditions, for example [["author",<,"Shakespeare"],["year","<=","2000"]] """ mask = np.ones(self.rec_array.shape[0]).astype(bool) for item in conditions: field,op,target = item field_op = "self.rec_array['%s'] %s '%s'" % (field,op,target) mask &= eval(field_op) selected_sales = self.rec_array['sales'][mask] return np.sum(selected_sales)  Based on this rec_array, given the data author = 4*["Shakespeare"]+ 2*["Dante"] book = 2*["Hamlet"] + 2*["Romeo"] + 2*["Inferno"] year = 3*["2000", "2001"] sales = [104.2, 99.0, 27.0, 19.0, 11.6, 12.6]  we create an instance test = rec_array() test.add_record(author,book,year,sales)  If, for example, you want the sales of Shakespeare's Romeo, you can simply do this test.get_view([["author","==","Shakespeare"],["book","==","Romeo"]])  the output is 46.0 or, you can also do test.get_view([["author","==","Shakespeare"],["year","<=","2000"]])  the output is 131.2
Question: I'm trying to draw in my mind the structure of the LSTMs and I don't understand what are the Kernel and Recurrent Kernel. According to this post in LSTMs section, the Kernel it's the four matrices that are multiplied by the inputs and Recurrent Kernel it's the four matrices that are multiplied by the hidden state, but, what are those 4 matrices in this diagram?  Are the gates? I was testing with this app how the unit variable of the code below affect the kernel, recurrent kernel and bias: model = Sequential() model.add(LSTM(unit = 1, input_shape=(1, look_back)))  with look_back = 1 it returns me that:  with unit = 2 it returns me this  With unit = 3 this  Testing with this values I could deducted this expressions  but I don't know how this works by inside. What does mean <1x(4u)> or <ux(4u)>? u = units

Answer: The kernels are basically the weights handled by the LSTM cell units = neurons, like the classic multilayer perceptron It is not shown in your diagram, but the input is a vector X with 1 or more values, and each value is sent in a neuron with its own weight w (the which we are going to learn with backpropagation) The four matrices are these (expressed as Wf, Wi, Wc, Wo):  When you add a neuron, you are adding other 4 weights\kernel So for your input vector X you have four matrix. And therefore 1 * 4 * units = kernel  Regarding the recurrent_kernel here you can find the answer. Basically in keras input and hidden state are not concatenated like in the example diagrams (W[ht-1, t]) but they are split and handled with other four matrices called U:  Because you have a hidden state x neuron, the weights U (all four U) are: units * (4 * units) = recurrent kernel  ht-1 comes in a recurrent way from all your neurons. Like in a multilayer perceptron, each output of a neuron goes in all the next recurrent layer neurons source: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Question: I'm trying to work through a tutorial at http://www.semspirit.com/artificial-intelligence/machine-learning/regression/support-vector-regression/support-vector-regression-in-python/ but there's no csv file included, so I'm using my own data. Here's the code so far: import numpy as np import pandas as pd from matplotlib import cm from matplotlib import pyplot as plt from mpl_toolkits.mplot3d import Axes3D from scipy import stats # Here's where I import my data; there's no csv file included in the tutorial import quasar_functions as qf dataset, datasetname, mags = qf.loaddata('sdss12') S = np.asarray(dataset[mags]) t = np.asarray(dataset['z']) t.reshape(-1,1) # Feature scaling from sklearn.preprocessing import StandardScaler as scale sc_S = scale() sc_t = scale() S2 = sc_S.fit_transform(S) t2 = sc_t.fit_transform(t)  The last line throws an error: ValueError: Expected 2D array, got 1D array instead: array=[4.17974 2.06468 5.46959 ... 0.41398 0.3672 1.9235 ]. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.  and yes, I've reshaped my target array t with t.reshape(-1,1) as shown here, here, here and here, but to no avail. Am I reshaping correctly? Here's all my variables: 

Answer: I am guessing you have a dataframe, so you need to reassign the variable t = t.reshape(-1,1): import pandas as pd dataset = pd.DataFrame(np.random.normal(2,1,(100,4)),columns=['z','x1','x2','x3']) mags = ['x1','x2','x3'] S = np.asarray(dataset[mags]) t = np.asarray(dataset['z']) t = t.reshape(-1,1) from sklearn.preprocessing import StandardScaler as scale sc_S = scale() sc_t = scale() S2 = sc_S.fit_transform(S) t2 = sc_t.fit_transform(t)  To check it works: np.mean(t2) 2.4646951146678477e-16
Question: Hello I am using sqlalchemy and pandas to process some data and then save everything to a table in an sql database. I am trying to find a quick easy and standardized way to check if a table exists in a database based on a table name. I have found the has_table() function but no working examples. Does anyone has something like give "engine connection" & "table name"-> return true or false if table exists

Answer: With SQLAlchemy 1.4+ you can call has_table by using an inspect object, like so: import sqlalchemy as sa # … engine = sa.create_engine(connection_uri) insp = sa.inspect(engine) print(insp.has_table("team", schema="dbo")) # True (or False, as the case may be)  For earlier versions of SQLAlchemy, see the other answer here.
Question: I'm writing some Python code on AWS Lambda. I haven't used AWS for a few months and I noticed that when I hit the TEST button the test no longer runs on the latest code that I've entered into the Lambda editor, even when I save the code. After some playing around I've found I need to press the DEPLOY button first, then my test runs fine. Is this right? Surely I should be able to test my code before deploying it. The official AWS documentation doesn't specify that deploying is necessry to run a test. 

Answer: Yes, The Deploy button is updating the code, The Test button just Invoke your lambda with a test Event (That can be configured)
Question: I am trying to use a TypeVar to indicate an init parameter as a certain type. But I am doing it wrong, or it might not even be possible. from typing import TypeVar T=TypeVar("T") class TestClass: def __init__(self,value:T): self._value=value a = TestClass(value=10) b = TestClass(value="abc") reveal_type(a._value) reveal_type(b._value)  I was hoping the reveal type of a._value would have been int and b._value to have been string. But they are both revealed as 'T`-1' Any help or insight appreciated! [EDIT] A little more expanded example. The BaseClass will be overridden and the actual type hint is provided by the overriding class. from typing import TypeVar T=TypeVar("T") class BaseClass: def __init__(self,value): self._value = value class Class1(BaseClass): def __init__(self,value:str): super().__init__(value) class Class2(BaseClass): def __init__(self,value:int): super().__init__(value) a = Class1("A value") b = Class2(10) reveal_type(a._value) reveal_type(b._value) 

Answer: By default, using a TypeVar restricts its scope only to the method/function in which it is used as an annotation. In order to scope a TypeVar to the instance and all methods/attributes, declare the class as Generic. from typing import TypeVar, Generic T=TypeVar("T") class BaseClass(Generic[T]): # Scope of `T` is the class: def __init__(self, value: T): # Providing some `T` on `__init__` self._value = value # defines the class' `T`  This allows declaring subclasses either as generic or as concrete. class Class1(BaseClass[str]): # "is a" BaseClass where `T = str` pass # No need to repeat ``__init__`` class ClassT(BaseClass[T]): # "is a" BaseClass where `T = T'` @property def value(self) -> T: return self._value reveal_type(Class1("Hello World")._value) # Revealed type is 'builtins.str*' reveal_type(Class1(b"Uh Oh!")._value) # error: Argument 1 to "Class1" has incompatible type "bytes"; expected "str" reveal_type(ClassT(42).value) # Revealed type is 'builtins.int*'
Question: I want to get the gradient of an embedding layer from a pytorch/huggingface model. Here's a minimal working example: from transformers import pipeline nlp = pipeline("zero-shot-classification", model="facebook/bart-large-mnli") responses = ["I'm having a great day!!"] hypothesis_template = 'This person feels {}' candidate_labels = ['happy', 'sad'] nlp(responses, candidate_labels, hypothesis_template=hypothesis_template)  I can extract the logits just fine, inputs = nlp._parse_and_tokenize(responses, candidate_labels, hypothesis_template) predictions = nlp.model(**inputs, return_dict=True, output_hidden_states=True) predictions['logits']  and the model returns a layer I'm interested in. I tried to retain the gradient and backprop with respect to a single logit I'm interested in: layer = predictions['encoder_hidden_states'][0] layer.retain_grad() predictions['logits'][0][2].backward(retain_graph=True)  However, layer.grad == None no matter what I try. The other named parameters of the model have their gradients computed, so I'm not sure what I'm doing wrong. How do I get the grad of the encoder_hidden_states?

Answer: I was also very surprised of this issue. Although I have never used the library I went down and did some debugging and found out that the issue is coming from the library transformers. The problem is comming from from this line : encoder_states = tuple(hidden_state.transpose(0, 1) for hidden_state in encoder_states)  If you comment it out, you will get the gradient just with some dimensions transposed. This issue is related to the fact that Pytorch Autograd does not do very well on inplace operations as mentioned here. So to recap the solution is to comment line 382 in modeling_bart.py. You will get the gradient with this shape T x B x C instead of B x T x C, but you can reshape it as you want later.
Question: Trying to install tensorflow to work with the GPU. Some documentation I see says tensorflow comes out of box with gpu support when detected. If so, what command can I use to see tensorflow is using my GPU? I have seen other documentation saying you need tensorflow-gpu installed. I have tried both but do not see how my GPU is being used?

Answer:  Go to command line and run Python  The following example lists the number of visible GPUs on the host. Docs import tensorflow as tf devices = tf.config.list_physical_devices('GPU') print(len(devices))  For CUDA Docs import tensorflow as tf tf.test.is_built_with_cuda()  Returns whether TensorFlow was built with CUDA (GPU) support. Docs You can check with following function too but it's deprecated** import tensorflow as tf tf.test.is_gpu_available()  Both returns True if your GPU is available
Question: I'm trying to run some old code from gaussian filter when I find out that python launcher gets stuck trying to do the imshow function. I tried:  Used Matplotlib to display a graph to see if the python launcher was the problem but no, graph showed up fine.  Remove process in between just to have the image read and display in fear that something in my code was breaking the launcher but no success.  Reinstalled opencv-python but no success.   Also saw one question like this in the google search but OP deleted it. Has anyone encounter this issue or has any fix for this? Example code: import cv2 as cv filename = 'chessboard.png' img = cv.imread(filename) cv.imshow('dst',img) cv.waitKey(0)  OS: MacOS Big Sur (11.0.1)

Answer: I resolved the issue with below steps:  Install the anaconda. Install the libraries needed. Run the script, there is an error appeared as below:   You might be loading two sets of Qt binaries into the same process. Check that all plugins are compiled against the right Qt binaries. Export DYLD_PRINT_LIBRARIES=1 and check that only one set of binaries are being loaded.   then I installed two libraies:  pip install opencv-python opencv-python-headless  Retry run the script, the image can be shown on the left top of the monitor.
Question: I want to make a Flutter app that uses a Python module. What are the options to integrate Python code, how can I marshal data between the 2 runtimes, can I make a standalone app that includes Python module that doesn't depend on local Python installation? My context: I have written Python code that can mark attendance using face recognition. It basically writes the (Name, Registration no. and entry time) to a .csv file of those whose faces match with the data(images of my friends and mine) that I've provided. I would like to create a desktop and/or mobile app that utilises that existing code.

Answer: I suggest you to use or convert your Python code as a Back-end code and Flutter code as Front End code. After that, your Flutter application can call the API via HTTP Requests and get data that it wants. Further reading about my suggestion:  Build a REST API with Python HTTP Requests Front-End Back-End : Introduction
Question: I am currently trying to install PyTorch (using the installation commands posted on PyTorch.org for pip) and when I run the command, my computer completely freezes. I tried this multiple times with the same result. I had to restart the computer a few times as well. On my current try, I put "-v" when trying to install and the pip seems to be stuck on "Looking up in the cache". I do not know how to proceed. As I mentioned, I've already tried this method multiple times. It worked the first time but did not install PyTorch as it gave me an error for not using "--user". Are there any solutions to this? EDIT: I did want to add that I have Python 3.8.6 (64bit)

Answer: After troubling shooting and a lot of restart, it seems like the issue came from when pip was trying to load a pre-downloaded file. Essentially, the first time I ran the installation command, pip downloaded files for pytorch but did not install pytorch due to some user privilege issue. The fix is to add --no-cache-dir in the pip install command. This will override the cache (pre-downloaded files) and download the files all over again. For me specifically, I also needed to add --user. In other words, the command went from pip install torch===1.7.0+cu110 torchvision===0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html  to pip --no-cache-dir install torch===1.7.0+cu110 torchvision===0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html --user
Question: Yesterday, my program was working perfectly fine. However, today it stopped working. I think that it may have something to do with the latest Mac OS update, as I had just installed it today. My testing code is shown below import matplotlib.pyplot as plt import numpy as np print("ehllow") zeroes = np.zeros((10,10)) plt.imshow(zeroes) plt.show()  Everything is going fine until I get to plt.show(). I had tried removing it, and the program ran smoothly, but once I added it back in I got the error  Segmentation fault: 11  and then it shows a python crash screen I have python version 3.7.6 64 bit for Mac.

Answer: Ok. Just for anyone wondering Just uninstalling and reinstalling the packages that were giving the error worked for me pip uninstall matplotlib pip install matplotlib
Question: I want to annotate a datetime field to know whether it is null. Something like: Table.objects.all().annotate(date_is_null=XXX(YYY)).values(date_is_null, .....)  What do I need to replace XXX(YYY) to check if the field is null? (I'm using .values().annotate() later for a Group By, so it must be in annotate first)

Answer: You can use an ExpressionWrapper to convert a Q object to an annotated BooleanField: from django.db.models import BooleanField, ExpressionWrapper, Q Table.objects.annotate( date_is_null=ExpressionWrapper( Q(date=None), output_field=BooleanField() ) ) Here Q(date=None) is thus the condition. If the DateTimeField has a different name, you should alter the condition accordingly. You can make the condition more verbose with: from django.db.models import BooleanField, ExpressionWrapper, Q Table.objects.annotate( date_is_null=ExpressionWrapper( Q(date__isnull=True), output_field=BooleanField() ) )
Question: I am trying to make a bot that welcomes people to the server that it's in and all of the code works except for the on_member_join event because it utilizes the new intents. I googled on how to work with the intents and tried everything but it still isn't working. intents = discord.Intents.default() intents.members = True client = commands.Bot(command_prefix=',', intents = intents)  How do I fix this?

Answer: intents = discord.Intents.default() intents.members = True client = commands.Bot(command_prefix=',', intents=intents)  You also have to enable privileged intents in the developer portal A Primer Gateway to Intents