eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to create a 2x2 plots for Anscombe data-set Loading Data-set and separating each class in data-set import seaborn as sns import matplotlib.pyplot as plt anscombe = sns.load_dataset('anscombe') dataset_1 = anscombe[anscombe['dataset'] == 'I'] dataset_2 = anscombe[anscombe['dataset'] == 'II'] dataset_3 = anscombe[anscombe['dataset'] == 'III'] dataset_4 = anscombe[anscombe['dataset'] == 'IV']  Creating a figure and dividing into 4 parts fig = plt.figure() axes_1 = fig.add_subplot(2,2,1) axes_2 = fig.add_subplot(2,2,2) axes_3 = fig.add_subplot(2,2,3) axes_4 = fig.add_subplot(2,2,4) axes_1.plot(dataset_1['x'], dataset_1['y'], 'o') axes_2.plot(dataset_2['x'], dataset_2['y'], 'o') axes_3.plot(dataset_3['x'], dataset_3['y'], 'o') axes_4.plot(dataset_4['x'], dataset_4['y'], 'o') axes_1.set_title('dataset_1') axes_2.set_title('dataset_2') axes_3.set_title('dataset_3') axes_4.set_title('dataset_4') fig.suptitle('Anscombe Data') fig.tight_layout()  The only output which i'm getting at each plot is [<matplotlib.lines.Line2D at 0x24592c94bc8>]  What am I doing wrong?

Answer: If you are working with a Jupyter Notebook then you can add the following line to the top cell where you call all your imports. The following command will render your graph %matplotlib inline
Question: I am trying to handle uncaught exceptions that occur when I run a thread. The python documentation at docs.python.org states that "threading.excepthook() can be overridden to control how uncaught exceptions raised by Thread.run() are handled." However, I can't seem to do it properly. It doesn't appear that my excepthook function is ever excecuted. What is the correct way to do this? import threading import time class MyThread(threading.Thread): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) def excepthook(self, *args, **kwargs): print("In excepthook") def error_soon(timeout): time.sleep(timeout) raise Exception("Time is up!") my_thread = MyThread(target=error_soon, args=(3,)) my_thread.start() time.sleep(7) 

Answer: threading.excepthook is a function that belongs to the threading module, not a method of the threading.Thread class, so you should override threading.excepthook instead with your own function: import threading import time def excepthook(args): print("In excepthook") threading.excepthook = excepthook class MyThread(threading.Thread): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) def error_soon(timeout): time.sleep(timeout) raise Exception("Time is up!") my_thread = MyThread(target=error_soon, args=(3,)) my_thread.start() time.sleep(7)
Question: I have two versions of Python installed on my Windows system. 3.7 is installed in C:\Python37 and 3.8 installed in Python 3.8. My PATH variables include the Python 3.7 executable. When I try to run 'virtualenv -p C:\Python38 ProjectFolder' I get the following error: RuntimeError: failed to query C:\Python38 with code 13 err: 'Access is denied'  This is true if I specify C:\Python37 as well. Isn't this supposed to create a virutalenv using the specified Python binaries? What am I doing wrong? Thanks in advance!

Answer: virtualenv -p C:\Python38\python.exe ProjectFolder  I.e. point -p to python executable, not to a directory.
Question: I have a python function foo with a while True loop inside. For background: It is expected do stream info from the web, do some writing and run indefinitely. The asserts test if the writing was done correctly. Clearly I need it to stop sometime, in order to test. What I did was to run via multirpocessing and introduce a timeout there, however when I see the test coverage, the function which ran through the multiprocessing, are not marked as covered. Question 1: Why does pytest now work this way? Question 2: How can I make this work? I was thinking it's probably because I technically exit the loop, so maybe pytest does not mark this as tested.... import time import multiprocessing def test_a_while_loop(): # Start through multiprocessing in order to have a timeout. p = multiprocessing.Process( target=foo name="Foo", ) try: p.start() # my timeout time.sleep(10) p.terminate() finally: # Cleanup. p.join() # Asserts below ...  More info  I looked into adding a decorator such as @pytest.mark.timeout(5), but that did not work and it stops the whole function, so I never get to the asserts. (as suggested here). If I don't find a way, I will just test the parts, but ideally I would like to find a way to test by breaking the loop. I know I can re-write my code in order to make it have a timeout, but that would mean changing the code to make it testable, which I don't think is a good design.  Mocks I have not tried (as suggested here), because I don't believe I can mock what I do, since it writes info from the web. I need to actually see the "original" working.  

Answer: Break out the functionality you want to test into a helper method. Test the helper method. def scrape_web_info(url): data = get_it(url) return data # In production: while True: scrape_web_info(...) # During test: def test_web_info(): assert scrape_web_info(...) == ...
Question: So I've been following Google's official tensorflow guide and trying to build a simple neural network using Keras. But when it comes to training the model, it does not use the entire dataset (with 60000 entries) and instead uses only 1875 entries for training. Any possible fix? import tensorflow as tf from tensorflow import keras import numpy as np fashion_mnist = keras.datasets.fashion_mnist (train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data() train_images = train_images / 255.0 test_images = test_images / 255.0 class_names = ['T-shirt', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle Boot'] model = keras.Sequential([ keras.layers.Flatten(input_shape=(28, 28)), keras.layers.Dense(128, activation='relu'), keras.layers.Dense(10) ]) model.compile(optimizer='adam', loss= tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=['accuracy']) model.fit(train_images, train_labels, epochs=10)  Output: Epoch 1/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3183 - accuracy: 0.8866 Epoch 2/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3169 - accuracy: 0.8873 Epoch 3/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3144 - accuracy: 0.8885 Epoch 4/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3130 - accuracy: 0.8885 Epoch 5/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3110 - accuracy: 0.8883 Epoch 6/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3090 - accuracy: 0.8888 Epoch 7/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3073 - accuracy: 0.8895 Epoch 8/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3057 - accuracy: 0.8900 Epoch 9/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3040 - accuracy: 0.8905 Epoch 10/10 1875/1875 [==============================] - 3s 2ms/step - loss: 0.3025 - accuracy: 0.8915 <tensorflow.python.keras.callbacks.History at 0x7fbe0e5aebe0>  Here's the original google colab notebook where I've been working on this: https://colab.research.google.com/drive/1NdtzXHEpiNnelcMaJeEm6zmp34JMcN38

Answer: The number 1875 shown during fitting the model is not the training samples; it is the number of batches. model.fit includes an optional argument batch_size, which, according to the documentation:  If unspecified, batch_size will default to 32.  So, what happens here is - you fit with the default batch size of 32 (since you have not specified anything different), so the total number of batches for your data is 60000/32 = 1875
Question: I have a cloud function with trigger type set to HTTP and also have a service account which is having permissions to Invoke the cloud function. I want to invoke the cloud function from a python script. I am using the following script to invoke the function: from google.oauth2 import service_account from google.auth.transport.urllib3 import AuthorizedHttp credentials = service_account.Credentials.from_service_account_file('/path/to/service-account-credentials.json') scoped_credentials = credentials.with_scopes(['https://www.googleapis.com/auth/cloud-platform']) authed_http = AuthorizedHttp(scoped_credentials) response = authed_http.request('GET', 'https://test-123456.cloudfunctions.net/my-cloud-function') print(response.status)  I am getting Unauthorized ( 401 ) error in response. Is this the correct way of invocation?

Answer: To be able to call your cloud function you need an ID Token against a Cloud Functions end point from google.oauth2 import service_account from google.auth.transport.requests import AuthorizedSession url = 'https://test-123456.cloudfunctions.net/my-cloud-function' creds = service_account.IDTokenCredentials.from_service_account_file( '/path/to/service-account-credentials.json', target_audience=url) authed_session = AuthorizedSession(creds) # make authenticated request and print the response, status_code resp = authed_session.get(url) print(resp.status_code) print(resp.text)
Question: This is my code and I try to calculate ROC score but I have a problem with ValueError: multiclass format is not supported. I'm already looking sci-kit learn but it doesn't help. In the end, I'm still have ValueError: multiclass format is not supported. This is my code from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import BaggingClassifier from sklearn.metrics import confusion_matrix,zero_one_loss from sklearn.metrics import classification_report,matthews_corrcoef,accuracy_score from sklearn.metrics import roc_auc_score, auc dtc = DecisionTreeClassifier() bc = BaggingClassifier(base_estimator=dtc, n_estimators=10, random_state=17) bc.fit(train_x, train_Y) pred_y = bc.predict(test_x) fprate, tprate, thresholds = roc_curve(test_Y, pred_y) results = confusion_matrix(test_Y, pred_y) error = zero_one_loss(test_Y, pred_y) roc_auc_score(test_Y, pred_y) FP = results.sum(axis=0) - np.diag(results) FN = results.sum(axis=1) - np.diag(results) TP = np.diag(results) TN = results.sum() - (FP + FN + TP) print('\n Time Processing: \n',time.process_time()) print('\n Confussion Matrix: \n', results) print('\n Zero-one classification loss: \n', error) print('\n True Positive: \n', TP) print('\n True Negative: \n', TN) print('\n False Positive: \n', FP) print('\n False Negative: \n', FN) print ('\n The Classification report:\n',classification_report(test_Y,pred_y, digits=6)) print ('MCC:', matthews_corrcoef(test_Y,pred_y)) print ('Accuracy:', accuracy_score(test_Y,pred_y)) print (auc(fprate, tprate)) print ('ROC Score:', roc_auc_score(test_Y,pred_y))  This is the traceback 

Answer: From the docs, roc_curve: "Note: this implementation is restricted to the binary classification task." Are your label classes (y) either 1 or 0? If not, I think you have to add the pos_label parameter to your roc_curve call. fprate, tprate, thresholds = roc_curve(test_Y, pred_y, pos_label='your_label')  Or: test_Y = your_test_y_array # these are either 1's or 0's fprate, tprate, thresholds = roc_curve(test_Y, pred_y)
Question: So, I'm trying to write a gzip file, actually from the net, but to simplify I wrote some very basic test. import gzip LINES = [b'I am a test line' for _ in range(100_000)] f = gzip.open('./test.text.gz', 'wb') for line in LINES: f.write(line) f.close()  It runs great, and I can see in Jupyter that it has created the test.txt.gz file in the directory listing. So I click on it expecting a whole host of garbage characters indicative of a binary file, like you would see in Notepad. However, instead I get this ... Error! test.text.gz is not UTF-8 encoded. Saving disabled. See console for more details  Which makes me think, oh my god, coding error, something is wrong with my encoding, my saving, can I save bytes ? Am I using the correct routines ?? And then spend 5 hours trying all combinations of code and modules.

Answer: The very simple answer to this is none of the above. This is a very misleading error message, especially when the code you've written was designed to save a binary file with a weird extension. What this actually means is ...  I HAVE NO IDEA HOW TO DISPLAY THIS DATA ! - Yours Jupyter  So, go to your File Explorer, Finder navigate to the just saved file and open it. Voila !! Everything worked exactly as planned, there is no error. Hope this saves other people many hours of debugging, and please Jupyter, change your error message.
Question: I am using google colab and would like to embed a gif. Does anyone know how to do this? I am using the code below and it is not animating the gif in the notebook. I would like the notebook to be interactive so that one can see what the code animates without having to run it.  I found many ways to do so that did not work in Google colab. The code and the GIF of interest is below. import matplotlib.pyplot as plt import matplotlib.image as mpimg img = mpimg.imread("/content/animationBrownianMotion2d.gif") plt.imshow(img)   I tried some of the solutions provided. import IPython from IPython.display import Image Image(filename='/content/animationBrownianMotion2d.gif')  and similarly import IPython from IPython.display import Image Image(filename='/content/animationBrownianMotion2d.gif',embed=True)  but got the error, --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-27-56bf6cd2b134> in <module>() 1 import IPython 2 from IPython.display import Image ----> 3 Image(filename='/content/animationBrownianMotion2d.gif',embed=True) 4 /usr/local/lib/python3.6/dist-packages/IPython/core/display.py in __init__(self, data, url, filename, format, embed, width, height, retina, unconfined, metadata) 1013 1014 if self.embed and self.format not in self._ACCEPTABLE_EMBEDDINGS: -> 1015 raise ValueError("Cannot embed the '%s' image format" % (self.format)) 1016 self.width = width 1017 self.height = height ValueError: Cannot embed the 'gif' image format  both times.

Answer: For external gif, you can use Jupyter's display as @knoop's answer. from IPython.display import Image Image(url='https://upload.wikimedia.org/wikipedia/commons/e/e3/Animhorse.gif')  But for a local file, you need to read the bytes and display it. !wget https://upload.wikimedia.org/wikipedia/commons/e/e3/Animhorse.gif Image(open('Animhorse.gif','rb').read())
Question: When using dask.to_parquet(df, filename) a subfolder filename is created and several files are written to that folder, whereas pandas.to_parquet(df, filename) writes exactly one file. Can I use dask's to_parquet (without using compute() to create a pandas df) to just write a single file?

Answer: Writing to a single file is very hard within a parallelism system. Sorry, such an option is not offered by Dask (nor probably any other parallel processing library). You could in theory perform the operation with a non-trivial amount of work on your part: you would need to iterate through the partitions of your dataframe, write to the target file (which you keep open) and accumulate the output row-groups into the final metadata footer of the file. I would know how to go about this with fastparquet, but that library is not being much developed any more.
Question: I have a a= torch.randn(28, 28, 8) and I want to swap dimensions (0, 1, 2) to (2, 0, 1). I tried b = a.transpose(2, 0, 1) , but I received this error: TypeError: transpose() received an invalid combination of arguments - got (int, int, int), but expected one of: * (name dim0, name dim1) * (int dim0, int dim1)  Is there any way that I can swap all at once?

Answer: You can use Pytorch's permute() function to swap all at once, >>>a = torch.randn(28, 28, 8) >>>b = a.permute(2, 0, 1) >>>b.shape torch.Size([8, 28, 28])
Question: I'm having problem with tensorflow. I want to use ImageDataGenerator, but I'm receiving error ModuleNotFoundError: No module named 'tf'. Not sure what is the problem. I added this tf.version to test will it work, and it shows the version of tensorflow.  import tensorflow as tf from tensorflow import keras print(tf.__version__) from tf.keras.preprocessing.image import ImageDataGenerator  When I run this code, I get this: 2.1.0 Traceback (most recent call last): File "q:/TF/Kamen papir maaze/rks.py", line 14, in <module> from tf.keras.preprocessing.image import ImageDataGenerator ModuleNotFoundError: No module named 'tf' 

Answer: The line  import tensorflow as tf  means you are importing tensorflow with an alias as tf to call it modules/functions.  You cannot use the alias to import other modules.  For your case, if you call directly  tf.keras.preprocessing.image.ImageDataGenerator(...)  then it will work.  or  you need to import the module with the right module name. i.e.  from tensorflow.keras.preprocessing.image import ImageDataGenerator
Question: I have a tensor inps, which has a size of [64, 161, 1] and I have some new data d which has a size of [64, 161]. How can I add d to inps such that the new size is [64, 161, 2]?

Answer: There is a cleaner way by using .unsqueeze() and torch.cat(), which makes direct use of the PyTorch interface: import torch # create two sample vectors inps = torch.randn([64, 161, 1]) d = torch.randn([64, 161]) # bring d into the same format, and then concatenate tensors new_inps = torch.cat((inps, d.unsqueeze(2)), dim=-1) print(new_inps.shape) # [64, 161, 2]  Essentially, unsqueezing the second dimension already brings the two tensors into the same shape; you just have to be careful to unsqueeze along the right dimension. Similarly, the concatenation is unfortunately named differently from the otherwise similarly named NumPy function, but behave the same. Note that instead of letting torch.cat figure out the dimension by providing dim=-1, you can also explicitly provide the dimension to concatenate along, in this case by replacing it with dim=2. Keep in mind the difference between concatenation and stacking, which is helpful for similar problems with tensor dimensions.
Question: I'm writing a python script where I need to open a ".txt" folder and analyse the text in there.  I have saved this ".txt" document in the same folder as my Python script.  But, when I go to open the file; file = open("words.txt",'r') I get the error: No such file or directory: 'words.txt'. I don't understand why this is happening? 

Answer: Maby it's because your current working directory is different from the directory your files are stored. Try giving the full path to the file file = open("<full_path>\words.txt",'r')
Question: Say I have one matrix and one vector as follows: import torch x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) y = torch.tensor([0, 2, 1])  is there a way to slice it x[y] so the result is: res = [1, 6, 8]  So basically I take the first element of y and take the element in x that corresponds to the first row and the elements' column.

Answer: You can specify the corresponding row index as: import torch x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) y = torch.tensor([0, 2, 1]) x[range(x.shape[0]), y] tensor([1, 6, 8])
Question: I'm using poetry for my Python package manager but I believe this would apply to any programming practices.  I've been doing this without knowing exactly what I'm doing, or how I should be doing.  When you use a package manager and install a new package, there's usually a .lock file change to keep your build deterministic.  Usually, I would commit these changes like: $ git add poetry.lock pyproject.toml $ git commit -m "Install packages: beautifulsoup4"  i.e, I make a commit every time I install/remove a package. I do it because I FEEL like this is what I should do, but I have 0 clue if this is actually a correct way to handle this.  Am I doing great? or is there any other specific convention & rules I should abide by to make it follow the best practices as close as possible? 

Answer: Disclaimer Please refer to this answer for the official stance and justification on the topic, which should be the top post in this thread. Below is my original answer, which I'll leave as it was.  The official recommendation of the poetry maintainers is to commit the lockfile if you develop a deployable application (as opposed to a library). That being said, my personal experience has been that it isn't necessary to commit lockfiles to VCS. The pyproject.toml file is the reference for correct build instructions, and the lockfile is the reference for a single successful deployment. Now, I don't know what the spec for poetry.lock is, but I had them backfire on me often enough during collaboration with colleagues in ways where only deleting them would fix the problem. A usual problem was that using different operation systems or python versions would lead to different lockfiles, and that just doesn't fly. I'll gladly let our CI build and persist an authoritative reference-lockfile to enable re-builds, but it still wouldn't be committed to the repository. If maintaining a shared lockfile is viable given your workflow - great! You avoid a step in your pipeline, have one less artifact to worry about, and cut down dramatically on build time (even a medium-size project can take minutes to do a full dependency-resolution). But as far as best practices go, I'd consider adding poetry.lock to the .gitignore a better practice than what you do, and only commit pyproject.toml changes when you add dependencies.
Question: I have my databricks python code in github. I setup a basic workflow to lint the python code using flake8. This fails because the names that are implicitly available to my script (like spark, sc, dbutils, getArgument etc) when it runs on databricks are not available when flake8 lints it outside databricks (in github ubuntu vm). How can I lint databricks notebooks in github using flake8? E.g. errors I get: test.py:1:1: F821 undefined name 'dbutils' test.py:3:11: F821 undefined name 'getArgument' test.py:5:1: F821 undefined name 'dbutils' test.py:7:11: F821 undefined name 'spark'  my notebook in github: dbutils.widgets.text("my_jdbcurl", "default my_jdbcurl") jdbcurl = getArgument("my_jdbcurl") dbutils.fs.ls(".") df_node = spark.read.format("jdbc")\ .option("driver", "org.mariadb.jdbc.Driver")\ .option("url", jdbcurl)\ .option("dbtable", "my_table")\ .option("user", "my_username")\ .option("password", "my_pswd")\ .load()  my .github/workflows/lint.yml on: pull_request: branches: [ master ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - uses: actions/setup-python@v1 with: python-version: 3.8 - run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Lint with flake8 run: | pip install flake8 flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics 

Answer: TL;DR Don't use the built-in variable dbutils in code that would need to run locally (IDE, Unit tests, ...) and in Databricks (production). Create your own instance of DBUtils class instead.  Here is what we ended up doing: Created a new dbk_utils.py from pyspark.sql import SparkSession def get_dbutils(spark: SparkSession): try: from pyspark.dbutils import DBUtils return DBUtils(spark) except ModuleNotFoundError: import IPython return IPython.get_ipython().user_ns["dbutils"]  And update the code that uses dbutils to use this utility: from dbk_utils import get_dbutils my_dbutils = get_dbutils() my_dbutils.widgets.text("my_jdbcurl", "default my_jdbcurl") my_dbutils.fs.ls(".") jdbcurl = my_dbutils.widgets.getArgument("my_jdbcurl") df_node = spark.read.format("jdbc")\ .option("driver", "org.mariadb.jdbc.Driver")\ .option("url", jdbcurl)\ .option("dbtable", "my_table")\ .option("user", "my_username")\ .option("password", "my_pswd")\ .load()  If you're trying to do unit testing as well, then check out:  How to mock an import and ModuleNotFoundError: No module named 'pyspark.dbutils'