eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using tf.random.set_seed to assure the reproducibility of my experiments but getting different results in terms of loss after training my model multiple times. I am monitoring the learning curve of each experiment using Tensorboard, but I am getting different values of loss and accuracy.

Answer: Providing the solution here (Answer Section), even though it is present in the Comment Section, for the benefit of the community. To reproduce same results, you can create function as below and pass seeds directly to the layers as mentioned Daniel in the comments. def reset_random_seeds(): os.environ['PYTHONHASHSEED']=str(2) tf.random.set_seed(2) np.random.seed(2) random.seed(2)  Please refer complete code in below, which reproduced same results import os ####*IMPORANT*: Have to do this line *before* importing tensorflow os.environ['PYTHONHASHSEED']=str(2) import tensorflow as tf import tensorflow.keras as keras import tensorflow.keras.layers import random import pandas as pd import numpy as np def reset_random_seeds(): os.environ['PYTHONHASHSEED']=str(2) tf.random.set_seed(2) np.random.seed(2) random.seed(2) #make some random data reset_random_seeds() NUM_ROWS = 1000 NUM_FEATURES = 10 random_data = np.random.normal(size=(NUM_ROWS, NUM_FEATURES)) df = pd.DataFrame(data=random_data, columns=['x_' + str(ii) for ii in range(NUM_FEATURES)]) y = df.sum(axis=1) + np.random.normal(size=(NUM_ROWS)) def run(x, y): reset_random_seeds() model = keras.Sequential([ keras.layers.Dense(40, input_dim=df.shape[1], activation='relu'), keras.layers.Dense(20, activation='relu'), keras.layers.Dense(10, activation='relu'), keras.layers.Dense(1, activation='linear') ]) NUM_EPOCHS = 100 model.compile(optimizer='adam', loss='mean_squared_error') model.fit(x, y, epochs=NUM_EPOCHS, verbose=0) predictions = model.predict(x).flatten() loss = model.evaluate(x, y) #This prints out the loss by side-effect #With Tensorflow 2.0 this is now reproducible! run(df, y) run(df, y) run(df, y)  Output: 32/32 [==============================] - 0s 2ms/step - loss: 0.5633 32/32 [==============================] - 0s 2ms/step - loss: 0.5633 32/32 [==============================] - 0s 2ms/step - loss: 0.5633
Question: I am running into the following error when trying to train this on this dataset. Since this is the configuration published in the paper, I am assuming I am doing something incredibly wrong. This error arrives on a different image every time I try to run training. C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed. Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1741, in <module> main() File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1735, in main globals = debugger.run(setup['file'], None, None, is_module) File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1135, in run pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "C:/Noam/Code/vision_course/hopenet/deep-head-pose/code/original_code_augmented/train_hopenet_with_validation_holdout.py", line 187, in <module> loss_reg_yaw = reg_criterion(yaw_predicted, label_yaw_cont) File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__ result = self.forward(*input, **kwargs) File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\modules\loss.py", line 431, in forward return F.mse_loss(input, target, reduction=self.reduction) File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\functional.py", line 2204, in mse_loss ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction)) RuntimeError: reduce failed to synchronize: cudaErrorAssert: device-side assert triggered   Any ideas?

Answer: This kind of error generally occurs when using NLLLoss or CrossEntropyLoss, and when your dataset has negative labels (or labels greater than the number of classes). That is also the exact error you are getting Assertion t >= 0 && t < n_classes failed.  This won't occur for MSELoss, but OP mentions that there is a CrossEntropyLoss somewhere and thus the error occurs (the program crashes asynchronously on some other line). The solution is to clean the dataset and ensure that t >= 0 && t < n_classes is satisfied (where t represents the label). Also, ensure that your network output is in the range 0 to 1 in case you use NLLLoss or BCELoss (then you require softmax or sigmoid activation respectively). Note that this is not required for CrossEntropyLoss or BCEWithLogitsLoss because they implement the activation function inside the loss function. (Thanks to @PouyaB for pointing out).
Question: I am stuck trying to run a very simple Python script, getting this error: qt.qpa.plugin: Could not find the Qt platform plugin "cocoa" in "" This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem. zsh: abort python3 mypuppy1.py  The script code is: import cv2 img = cv2.imread('00-puppy.jpg') while True: cv2.imshow('Puppy',img) if cv2.waitKey(1) & 0xFF == 27: break cv2.destroyAllWindows()  However this Notebook code works in JupyterLab: import cv2 img = cv2.imread('00-puppy.jpg') cv2.imshow('Puppy', img) cv2.waitKey()  I am on macOS, using Anaconda and JupyterLab. I would appreciate any help with this issue. Thanks!

Answer: For me, it worked by using a opencv-python version prior to 4.2 version that just got released. The new version (4.2.0.32) released on Feb 2, 2020 seems to have caused this breaking change and probably expects to find Qt at a specific location (Users/ directory) as pointed by other answers.  You can try either manually installed from qt.io as suggested and making sure you get a .qt directory under yours Users directory, or you can use version 4.1.2.30, which works like charm without doing anything else. It works for opencv-contrib-python too.
Question: I have a model that, based on certain conditions, has some unconnected gradients, and this is exactly what I want. But Tensorflow is printing out a Warning every time it encounters the unconnected gradient. WARNING:tensorflow:Gradients do not exist for variables  Is there any way to only suppress this specific warning? I don't want to blindly suppress all warnings since there might be unexpected (and potentially useful) warnings in the future as I'm still working on my model.

Answer: Kinda hacky way: gradients = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients([ (grad, var) for (grad, var) in zip(gradients, model.trainable_variables) if grad is not None ])
Question: The database of my app is populated and kept syncd with external data sources. I have an abstract model from which all the models of my Django 2.2 app derives, defined as follow: class CommonModel(models.Model): # Auto-generated by Django, but included in this example for clarity. # id = models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID') ORIGIN_SOURCEA = '1' ORIGIN_SOURCEB = '2' ORIGIN_CHOICES = [ (ORIGIN_SOURCEA, 'Source A'), (ORIGIN_SOURCEB, 'Source B'), ] object_origin = models.IntegerField(choices=ORIGIN_CHOICES) object_id = models.IntegerField() class A(CommonModel): some_stuff = models.CharField() class B(CommonModel): other_stuff = models.IntegerField() to_a_fk = models.ForeignKey("myapp.A", on_delete=models.CASCADE) class C(CommonModel): more_stuff = models.CharField() b_m2m = models.ManyToManyField("myapp.B")  The object_id field can't be set as unique since each data source I use in my app may have an object with an object_id = 1. Hence the need to track down the origin of the object, by the field object_origin. Unfortunately, Django's ORM doesn't support more-than-one columns foreign keys. Problem Whilst keeping the auto-generated primary key in the database (id), I would like to make my foreign key and many-to-many relations happen on both object_id and object_origin fields instead of the primary key id. What I've tried I thought about doing something like this: class CommonModel(models.Model): # Auto-generated by Django, but included in this example for clarity. # id = models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID') ORIGIN_SOURCEA = '1' ORIGIN_SOURCEB = '2' ORIGIN_CHOICES = [ (ORIGIN_SOURCEA, 'Source A'), (ORIGIN_SOURCEB, 'Source B'), ] object_origin = models.IntegerField(choices=ORIGIN_CHOICES) object_id = models.IntegerField() def _get_composed_object_origin_id(self): return f"{self.object_origin}:{self.object_id}" composed_object_origin_id = property(_get_composed_object_origin_id) class A(CommonModel): some_stuff = models.CharField() class B(CommonModel): other_stuff = models.IntegerField() to_a_fk = models.ForeignKey("myapp.A", to_field="composed_object_origin_id", on_delete=models.CASCADE)  But Django complains about it: myapp.B.to_a_fk: (fields.E312) The to_field 'composed_object_origin_id' doesn't exist on the related model 'myapp.A'. And it sounds legit, Django excepted the filed given to to_field to be a database field. But there is no need to add a new field to my CommonModel since composed_object_type_id is built uisng two non-nullable fields... 

Answer: You mentioned in your comment in the other answer that object_id is not unique but it is unique in combination with object_type, so could you use a unique_together in the metaclass? i.e class CommonModel(models.Model): object_type = models.IntegerField() object_id = models.IntegerField() class Meta: unique_together = ( ("object_type", "object_id"), )
Question: I am having a problem that I can not use cv2.imshow() because of following error message  qt.qpa.plugin: Could not find the Qt platform plugin "cocoa" in "" This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.  Last Macbook I was using initially did not have QT so I have no idea how should I deal with it. Any ideas?

Answer: I had the same issue after updating opencv - python to 4.2.0.32. Uninstall opencv-python and install the lower version (e.g pip install opencv-python==4.1.0.25) solves this issue.
Question: I'm using pydantic 1.3 to validate models for an API I am writing. Is it common/good practice to include arbitrary methods in a class that inherits from pydantic.BaseModel? I need some helper methods associated with the objects and I am trying to decide whether I need a "handler" class. These models are being converted to JSON and sent to a restful service that I am also writing. My model looks like this: class Foo(pydantic.BaseModel): name: str bar: int baz: int  Is it poor practice to do something like: class Foo(pydantic.BaseModel): name: str bar: int baz: int def add_one(self): self.bar += 1  It makes some sense to me, but I can't find an example of anyone doing this.

Answer: Yes, it's fine. We should probably document it. The only problem comes when you have a field name which conflicts with the method, but that's not a problem if you know what your data looks like. Also, it's possible to over object orient your code, but you're a long way from that.
Question: This is my code to save web_cam streaming. It is working but the problem with output video file. import numpy as np import cv2 cap = cv2.VideoCapture(0) # Define the codec and create VideoWriter object #fourcc = cv2.cv.CV_FOURCC(*'DIVX') #out = cv2.VideoWriter('output.avi',fourcc, 20.0, (640,480)) out = cv2.VideoWriter('output.avi', -1, 20.0, (640,480)) while(cap.isOpened()): ret, frame = cap.read() if ret==True: frame = cv2.flip(frame,0) # write the flipped frame out.write(frame) cv2.imshow('frame',frame) if cv2.waitKey(1) & 0xFF == ord('q'): break else: break # Release everything if job is finished cap.release() out.release() cv2.destroyAllWindows() 

Answer: The output file is corrupted because of the wrong frame rate and frame resolution. Using this code :  out = cv2.VideoWriter('output.avi', -1, 20.0, (640,480))  We set the fps/frame rate per second 20. Which was not correct. Also, the frame width and height was wrong. I solved by getting fps, width, height from the captured web_cam profile.  cap = cv2.VideoCapture(0) #web-cam capture fps = cap.get(cv2.CAP_PROP_FPS) width = cap.get(cv2.CAP_PROP_FRAME_WIDTH) # float height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT) # float out = cv2.VideoWriter('output.avi', -1,fps, (int(width), int(height)))
Question: I am trying to read Prometheus metrics (the cpu and memory values) of a POD in kubernetes. I have Prometheus install and everything is up using local host 'http://localhost:9090/. I used the following code to read the CPU and memory of a pod but I have an error results = response.json()['data']['result'] , No JSON object could be decoded . IS anyone can help please ?  import datetime import time import requests PROMETHEUS = 'http://localhost:9090/' end_of_month = datetime.datetime.today().replace(day=1).date() last_day = end_of_month - datetime.timedelta(days=1) duration = '[' + str(last_day.day) + 'd]' response = requests.get(PROMETHEUS + '/metrics', params={ 'query': 'sum by (job)(increase(process_cpu_seconds_total' + duration + '))', 'time': time.mktime(end_of_month.timetuple())}) results = response.json()['data']['result'] print('{:%B %Y}:'.format(last_day)) for result in results: print(' {metric}: {value[1]}'.format(**result)) 

Answer: The code looks true, However,the query in your response command is wrong . the true formate is :  response =requests.get(PROMETHEUS + '/api/v1/query', params={'query': 'container_cpu_user_seconds_total'})  you can change "container_cpu_user_seconds_total" to any query that you want to read. .. good luck
Question: Using apache airflow, I created some DAGS, some of which do not run on a schedule. I'm trying to find a way that I can trigger a run for a specific DAG from within a Python script. Is this possible? How can I do?  EDIT --- The python script will be running from a different project from the project where all my DAGS are located

Answer: You have a variety of options when it comes to triggering Airflow DAG runs. Using Python The airflow python package provides a local client you can use for triggering a dag from within a python script. For example: from airflow.api.client.local_client import Client c = Client(None, None) c.trigger_dag(dag_id='test_dag_id', run_id='test_run_id', conf={})  Using the Airflow CLI You can trigger dags in airflow manually using the Airflow CLI. More info on how to use the CLI to trigger DAGs can be found here. Using the Airflow REST API You can also use the Airflow REST api to trigger DAG runs. More info on that here.  The first option from within python might work for you best (it's also how I've personally done it in the past). But you could theoretically use a subprocess to interact with the CLI from python, or a library like requests to interact with the REST API from within Python.
Question: I 'm using Seaborn in a Jupyter notebook to plot histograms like this: import numpy as np import pandas as pd from pandas import DataFrame import matplotlib.pyplot as plt import seaborn as sns %matplotlib inline df = pd.read_csv('CTG.csv', sep=',') sns.distplot(df['LBE'])  I have an array of columns with values that I want to plot histogram for and I tried plotting a histogram for each of them: continous = ['b', 'e', 'LBE', 'LB', 'AC'] for column in continous: sns.distplot(df[column])  And I get this result - only one plot with (presumably) all histograms:  My desired result is multiple histograms that looks like this (one for each variable):  How can I do this?

Answer: Insert plt.figure() before each call to sns.distplot() . Here's an example with plt.figure():  Here's an example without plt.figure():  Complete code: # imports import numpy as np import pandas as pd import seaborn as sns import matplotlib.pyplot as plt plt.rcParams['figure.figsize'] = [6, 2] %matplotlib inline # sample time series data np.random.seed(123) df = pd.DataFrame(np.random.randint(-10,12,size=(300, 4)), columns=list('ABCD')) datelist = pd.date_range(pd.datetime(2014, 7, 1).strftime('%Y-%m-%d'), periods=300).tolist() df['dates'] = datelist df = df.set_index(['dates']) df.index = pd.to_datetime(df.index) df.iloc[0]=0 df=df.cumsum() # create distplots for column in df.columns: plt.figure() # <==================== here! sns.distplot(df[column])
Question: I've got a Python project aws-ssm-tools that uses setup.py for packaging. It comes with 3 scripts: ssm-tunnel, ssm-session and ssm-copy. It can be installed with pip install aws-ssm-tools and puts the scripts to ~/.local/bin/. However when I try to install it with pipx it fails: ~ $ pipx install aws-ssm-tools No apps associated with package aws-ssm-tools. Try again with '--include-deps' to include apps of dependent packages, which are listed above. If you are attempting to install a library, pipx should not be used. Consider using pip or a similar tool instead.  I have the scripts specified in setup.py: SCRIPTS = [ 'ssm-session', 'ssm-copy', 'ssm-tunnel', 'ssm-tunnel-updown.dns-example', ] # ... setup( name="aws-ssm-tools", version=VERSION, packages=find_packages(), scripts=SCRIPTS+[ 'ssm-tunnel-agent' ], # ... )  What else do I have to do to make pipx happy?

Answer: As stated in the pipx documentation chapter "How pipx works", section "Developing for pix", the project requires setuptools entry_points. According to the content of your question, it seems that the target project uses scripts, they are similar in purpose to entry-points but pipx does not look for those and does not expose them.
Question: I'm trying to compare two images, one a .png and the other a .jpg. So I need to convert the .png file to a .jpg to get closer values for SSIM. Below is the code that I've tried, but I'm getting this error:  AttributeError: 'tuple' object has no attribute 'dtype'  image2 = imread(thisPath + caption) image2 = io.imsave("jpgtest.jpg", (76, 59)) image2 = cv2.cvtColor(image2, cv2.COLOR_BGR2GRAY) image2 = resize(image2, (76, 59)) imshow("is it a jpg", image2) cv2.waitKey() 

Answer: Before demonstrating how to convert an image from .png to .jpg format, I want to point out that you should be consistent on the library that you use. Currently, you're mixing scikit-image with opencv. It's best to choose one library and stick with it instead of reading in an image with scikit and then converting to grayscale with opencv.  To convert a .png to .jpg image using OpenCV, you can use cv2.imwrite. Note with .jpg or .jpeg format, to maintain the highest quality, you must specify the quality value from [0..100] (default value is 95). Simply do this: import cv2 # Load .png image image = cv2.imread('image.png') # Save .jpg image cv2.imwrite('image.jpg', image, [int(cv2.IMWRITE_JPEG_QUALITY), 100])
Question: I am building a web app with Django. I would like to be able to store final user settings for my app. For now I am using a model Settings that contains all what I need for the app.  It seems ok but I don't think that's the right way to do it because I could have a second row in my database and I don't want it. I though about using a file but it is not as easy to manage as a table in a database. Could someone help me with that please ? Thank you ! :) Louison

Answer: If I understand your question correctly, you're interested in managing settings for your django app through an editable, easy to update interface. A popular existing approach to this is implemented by django-constance. It can use your database or redis to store the settings, and it makes them editable through the django admin. You can look at the documentation here.  Ultimately, the approach of using a database table to store configuration settings is fine. The issue of having one row can feel odd though, so django-constance's approach is to use a key/value design where each individual configuration key has its own row in the table.
Question: The following script reads a video with OpenCV, applies a transformation to each frame and attempts to write it with ffmpeg. My problem is, that I don't get ffmpeg working with the subprocess module. I always get the error BrokenPipeError: [Errno 32] Broken pipe in the line where I try to write to stdin. Why is that, what am I doing wrong? # Open input video with OpenCV video_in = cv.VideoCapture(src_video_path) frame_width = int(video_in.get(cv.CAP_PROP_FRAME_WIDTH)) frame_height = int(video_in.get(cv.CAP_PROP_FRAME_HEIGHT)) fps = video_in.get(cv.CAP_PROP_FPS) frame_count = int(video_in.get(cv.CAP_PROP_FRAME_COUNT)) bitrate = bitrate * 4096 * 2160 / (frame_width * frame_height) # Process video in ffmpeg pipe # See http://zulko.github.io/blog/2013/09/27/read-and-write-video-frames-in-python-using-ffmpeg/ command = ['ffmpeg', '-loglevel', 'error', '-y', # Input '-f', 'rawvideo', '-vcodec', 'rawvideo' '-pix_fmt', 'bgr24', '-s', str(frame_width) + 'x' + str(frame_height), '-r', str(fps), # Output '-i', '-', '-an', '-vcodec', 'h264', '-r', str(fps), '-b:v', str(bitrate) + 'M', '-pix_fmt', 'bgr24', dst_video_path ] pipe = sp.Popen(command, stdin=sp.PIPE) for i_frame in range(frame_count): ret, frame = video_in.read() if ret: warped_frame = cv.warpPerspective(frame, homography, (frame_width, frame_height)) pipe.stdin.write(warped_frame.astype(np.uint8).tobytes()) else: print('Stopped early.') break print('Done!') 

Answer: There is a missing comma after '-vcodec', 'rawvideo'!!!  Took me about an hour to notice...  You should also close stdin and wait before print('Done!'):  pipe.stdin.close() pipe.wait()
Question: I'm currently working on an app using Django 2.2 with djangorestframework 3.9.2. I am aware that Django itself provides protection against SQL Injection or in in the context of displaying content in django templates (XSS), but I've noticed that while I use Django REST API, all the CharFields in my models are not sanitized automatically. Note: this question does not apply to django templates. E.g. having a direct messages model (message/models.py): class Message(models.Model): sender = models.ForeignKey(...) receiver = models.ForeignKey(...) message = models.CharField(max_length=1200) timestamp = models.DateTimeField(...) is_read = models.BooleanField(default=False)  Actually does not prevent anyone from sending a message with content <script>alert("Hello there");</script>. It will be saved in database and returned by the REST API as is, allowing to remotely run any JS script (basically a Cross Site Scripting). Is this an expected behavior? How can this be prevented?

Answer: You can use escape() method inside serializer's validation: from django.utils.html import escape class MySerializer: def validate_myfield(self, value): return escape(value)
Question: Is there any way to use moving plot without ffmpeg? import matplotlib.animation as animation from IPython.display import HTML fig, ax = plt.subplots(figsize=(15, 8)) animator = animation.FuncAnimation(fig, draw_barchart, frames=range(1968, 2019)) HTML(animator.to_jshtml()) animator.save('dynamic_images.mp4')  My code is above, I am getting the key error.mp4', ValueError: unknown file extension: .mp4 I tried installing conda install -c conda-forge ffmpeg end up with SSL issue  Is there any way to use moving plot without ffmpeg As like error throwing is there any way to use 'matplotlib.animation.PillowWriter'  Disclaimer : I went through the link https://www.wikihow.com/Install-FFmpeg-on-Windows but the URL is blocked by the IT team

Answer: You can save animated plot as .gif with use of celluloid library: from matplotlib import pyplot as plt from celluloid import Camera import numpy as np # create figure object fig = plt.figure() # load axis box ax = plt.axes() # set axis limit ax.set_ylim(0, 1) ax.set_xlim(0, 10) camera = Camera(fig) for i in range(10): ax.scatter(i, np.random.random()) plt.pause(0.1) camera.snap() animation = camera.animate() animation.save('animation.gif', writer='PillowWriter', fps=2)  Output: