eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm using boto3 (aws sdk for python) to analyze a document (a pdf) to get the form key:value pairs. import boto3 def process_text_analysis(bucket, document): # Get the document from S3 s3_connection = boto3.resource('s3') s3_object = s3_connection.Object(bucket, document) s3_response = s3_object.get() # Analyze the document client = boto3.client('textract') response = client.analyze_document(Document={'S3Object': {'Bucket': bucket, 'Name': document}}, FeatureTypes=["FORMS"]) process_text_analysis('francismorgan-01', '709 Privado M SURESTE.pdf')  I have followed the documentation for AWS using Analyze Document and when I run my function I get the error. botocore.errorfactory.UnsupportedDocumentException: An error occurred (UnsupportedDocumentException) when calling the AnalyzeDocument operation: Request has unsupported document format  Am I missing something? 

Answer: AnalyzeDocument is a synchronous API that only supports PNG or JPG images. Since you want to work with PDF files, then you'll need to use Amazon Textract Asynchronous API e.g StartDocumentAnalysis, StartDocumentTextDetection
Question: I need to get file info (path, size, dates, etc) and save it in a txt but I don't know where or how to do it. This is what I have: ruta = "FolderPath" os.listdir(path=ruta) miArchivo = open("TxtPath","w") def getListOfFiles(ruta): listOfFile = os.listdir(ruta) allFiles = list() for entry in listOfFile: fullPath = os.path.join(ruta, entry) if os.path.isdir(fullPath): allFiles = allFiles + getListOfFiles(fullPath) else: allFiles.append(fullPath) return allFiles listOfFiles = getListOfFiles(ruta) for elem in listOfFiles: print(elem) print("\n") miArchivo.write("%s\n" % (elem)) miArchivo.close()  The output is (only path, no other info):  What I want is: V:\1111111\222222222\333333333\444444444\5555555555\66666666\Folder\File name -- size -- modification date and so on

Answer: https://docs.python.org/2.7/library/os.path.html#module-os.path  os.path.getsize(path) # size in bytes os.path.ctime(path) # time of last metadata change; it's a bit OS specific.  Here's a rewrite of your program. I did this:  Reformatted with autopep8 for better readability. (That's something you can install to prettify your code your code. But IDEs such as PyCharm Community Edition can help you to do the same, in addition to helping you with code completion and a GUI debugger.)  Made your getListofFiles() return a list of tuples. There are three elements in each one; the filename, the size, and the timestamp of the file, which appears to be what's known as an epoch time (time in seconds since 1970; you will have to go through python documentation on dates and times).  The tuples is written to your text file in a .csv style format (but note there are modules to do the same in a much better way).  Rewritten code: import os def getListOfFiles(ruta): listOfFile = os.listdir(ruta) allFiles = list() for entry in listOfFile: fullPath = os.path.join(ruta, entry) if os.path.isdir(fullPath): allFiles = allFiles + getListOfFiles(fullPath) else: print('getting size of fullPath: ' + fullPath) size = os.path.getsize(fullPath) ctime = os.path.getctime(fullPath) item = (fullPath, size, ctime) allFiles.append(item) return allFiles ruta = "FolderPath" miArchivo = open("TxtPath", "w") listOfFiles = getListOfFiles(ruta) for elem in listOfFiles: miArchivo.write("%s,%s,%s\n" % (elem[0], elem[1], elem[2])) miArchivo.close()  Now it does this. my-MBP:verynew macbookuser$ python verynew.py; cat TxtPath getting size of fullPath: FolderPath/dir2/file2 getting size of fullPath: FolderPath/dir2/file1 getting size of fullPath: FolderPath/dir1/file1 FolderPath/dir2/file2,3,1583242888.4 FolderPath/dir2/file1,1,1583242490.17 FolderPath/dir1/file1,1,1583242490.17 my-MBP:verynew macbookuser$
Question: The situation I use Anaconda 3 on Windows 10. I have a Visual Studio Code workspace (my_workspace) than contains a Jupyter notebook (my_notebook.ipynb). VSCode has the Python extension installed. The file my_workspace/settings.json contains: { "python.pythonPath": "C:\\Users\\Me\\Anaconda3\\envs\\my_env\\python.exe" }  my_env is an existing Anaconda environment. I can activate it and work with it in a shell, and if I run jupyter lab in such a shell, the code inside the notebooks can import my_env's packages as expected. If I open my_workspace in VSCode, then my_notebook.ipynb in a tab, my_env is also mentioned in VSCode's status bar ("Python 3.7.6 64-bit ('my_env': conda)"), and my_env is automatically activated when I open a PowerShell prompt in VSCode's console (I ran conda init once). The problem When the notebook is opened in VSCode, the Jupyter kernel seems to use the base environment's Python interpreter instead of the one in my_env. When importing a package installed in my_env, but not in base, I get this error: >>> import keras Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'keras'  This happens for all packages, not just keras. In the notebook tab in VSCode, if I click on the interpreter's name in the top-right corner, then choose the correct interpreter (the one in my_env), then the notebook runs correctly in my_env. But I have to do this every time I re-open VSCode. How to make the default kernel respect the environment chosen in settings.json?

Answer: I think there is no parameter right now to control that in the settings.json. I had similar problems with the environments in which the notebook is launched and I was able to fix this modifying the kernelspec section in the IPython notebook.  Basically, open the notebook as a JSON file and remove the kernelspec section. When the notebook is launched from vscode, that part will be filled with the default python environment kernel for the workspace. In my case, is filled with the pipenv environment.
Question: https://docs.python.org/3/using/cmdline.html This is the option documentation. but it doesn't provide me any useful message I want to execute code in this way  python -c "def hello():\n print('hello world')"  the error message PS C:\Users\Administrator> python -c "def hello():\n print('hello world')" File "<string>", line 1 def hello():\n print('hello world') ^ SyntaxError: unexpected character after line continuation character  it works on Linux, but not windows.  Hope you can give a full fixed command ~~~  Another question connected with this one \n problem when using javascript to exec "python -c "

Answer: Try backtick instead of backslash. Error: PS C:\Users\me> python -c "def hello():\n print('hello world')" File "<string>", line 1 def hello():\n print('hello world') ^ SyntaxError: unexpected character after line continuation character PS C:\Users\me>  Ok: PS C:\Users\me> python -c "def hello():`n print('hello world')" PS C:\Users\me>  Useful: PS C:\Users\me> python -c "def hello():`n print('hello world')`nhello()" hello world PS C:\Users\me>  Just echoing to see it: PS C:\Users\me> echo "def hello():`n print('hello world')`nhello()" def hello(): print('hello world') hello() PS C:\Users\me>  See PowerTip: New Lines with PowerShell
Question: I have two datasets that I need to validate against. All records should match. I am having trouble in determining how to iterate through each different column.  import pandas as pd import numpy as np df = pd.DataFrame([['charlie', 'charlie', 'beta', 'cappa'], ['charlie', 'charlie', 'beta', 'delta'], ['charlie', 'charlie', 'beta', 'beta']], columns=['A_1', 'A_2','B_1','B_2']) df.head() Out[83]: A_1 A_2 B_1 B_2 0 charlie charlie beta cappa 1 charlie charlie beta delta 2 charlie charlie beta beta  For example, in the above code, I want to compare A_1 to A_2, and B_1 to B_2, to return a new column, A_check and B_check respectively, that return True if A_1 matches A_2 as the A_Check for instance.  Something like this:  df['B_check'] = np.where((df['B_1'] == df['B_2']), 'True', 'False') df_subset = df[df['B_check']=='False']  But iterable across any given column names, where columns that need to be checked against will always have the same name before the underscore and always have 1 or 2 after the underscore.  Ultimately, the actual task has multiple data frames with varying columns to check, as well as varying numbers of columns to check. The output I am ultimately going for is a data frame that shows all the records that were false for any particular column check. 

Answer: With a bit more comprehensive regex: from itertools import groupby import re for k, cols in groupby(sorted(df.columns), lambda x: x[:-2] if re.match(".+_(1|2)$", x) else None): cols=list(cols) if(len(cols)==2 and k): df[f"{k}_check"]=df[cols[0]].eq(df[cols[1]])  It will pair together only columns which name ends up with _1 and _2 regardless what you have before in their names, calculating _check only if there are 2- _1 and _2 (assuming you don't have 2 columns with the same name). For the sample data:  A_1 A_2 B_1 B_2 A_check B_check 0 charlie charlie beta cappa True False 1 charlie charlie beta delta True False 2 charlie charlie beta beta True True
Question: Problem I came across this code in Object Oriented Programming by Dusty Phillips (simplified for brevity) and I unsure about a specific part of this definition. class A: def __init__(self, a, **kwargs): super().__init__(**kwargs) self.a = a class B: def __init__(self, b, **kwargs): super().__init__(**kwargs) self.b = b class C(A, B): def __init__(self, c, **kwargs): super().__init__(**kwargs) self.c = c   Questions  Since the method resolution order is (__main__.C, __main__.A, __main__.B, object), could class B be defined in the following way instead?  class B: def __init__(self, b): self.b = b   Isn't super().__init__(**kwargs) in class B redundant, since any surplus kwargs passed to C will be passed to object, raising?   TypeError: object.__init__() takes exactly one argument (the instance to initialize)   Is this a safeguard for if C was defined as class C(B, A) instead of class C(A, B)? 

Answer:  Since the method resolution order is (__main__.C, __main__.A, __main__.B, object), could class B be defined in the following way instead?  No, because then this would fail: class D: def __init__(self, d, **kwargs): self.d = d super().__init__(**kwargs) class E(C, D): def __init__(self, e, **kwargs): self.e = e super().__init__(**kwargs)  The MRO of E is (E, C, A, B, D, object), so B must call super().__init__ otherwise D.__init__ won't be called.  Isn't super().__init__(**kwargs) in class B redundant, since any surplus kwargs passed to C will be passed to object, raising?  No, because the surplus kwargs will go to D.__init__ in the above example. But even without that, it is not redundant to raise an error when you call a constructor with too many arguments; it is desirable to have an error message informing you about your incorrect code, rather than for the mistake to go undetected.  Is this a safeguard for if C was defined as class C(B, A) instead of class C(A, B)?  In some sense, sure; but really it's a safeguard for B occurring in any class hierarchy, so long as the other classes in the hierarchy follow the same rule of calling super().__init__(**kwargs).
Question: Using pandas v1.0.1 and numpy 1.18.1, I want to calculate the rolling mean and std with different window sizes on a time series. In the data I am working with, the values can be constant for some subsequent points such that - depending on the window size - the rolling mean might be equal to all the values in the window and the corresponding std is expected to be 0. However, I see a different behavior using the same df depending on the window size. MWE: for window in [3,5]: values = [1234.0, 4567.0, 6800.0, 6810.0, 6821.0, 6820.0, 6820.0, 6820.0, 6820.0, 6820.0, 6820.0] df = pd.DataFrame(values, columns=['values']) df.loc[:, 'mean'] = df.rolling(window, min_periods=1).mean() df.loc[:, 'std'] = df.rolling(window, min_periods=1).std(ddof=0) print(df.info()) print(f'window: {window}') print(df) print('non-rolling result:', df['values'].iloc[len(df.index)-window:].values.std()) print('')  Output: window: 3 values mean std 0 1234.0 1234.000000 0.000000 1 4567.0 2900.500000 1666.500000 2 6800.0 4200.333333 2287.053757 3 6810.0 6059.000000 1055.011216 4 6821.0 6810.333333 8.576454 5 6820.0 6817.000000 4.966555 6 6820.0 6820.333333 0.471405 7 6820.0 6820.000000 0.000000 8 6820.0 6820.000000 0.000000 9 6820.0 6820.000000 0.000000 10 6820.0 6820.000000 0.000000 non-rolling result: 0.0 window: 5 values mean std 0 1234.0 1234.000000 0.000000 1 4567.0 2900.500000 1666.500000 2 6800.0 4200.333333 2287.053757 3 6810.0 4852.750000 2280.329732 4 6821.0 5246.400000 2186.267193 5 6820.0 6363.600000 898.332366 6 6820.0 6814.200000 8.158431 7 6820.0 6818.200000 4.118252 8 6820.0 6820.200000 0.400000 9 6820.0 6820.000000 0.000021 10 6820.0 6820.000000 0.000021 non-rolling result: 0.0  As expected, the std is 0 for idx 7,8,9,10 using a window size of 3. For a window size of 5, I would expect idx 9 and 10 to yield 0. However, the result is different from 0. If I calculate the std 'manually' for the last window of each window size (using idxs 8,9,10 and 6,7,8,9,10, respectively), I get the expected result of 0 for both cases. Does anybody have an idea what could be the issue here? Any numerical caveats?

Answer: It seems that implementation of std() in pd.rolling prefers high performance over numerical accuracy. However You can apply np version of standard deviation: df.loc[:, 'std'] = df.rolling(window, min_periods=1).apply(np.std)  Result:  values std 0 1234.0 0.000000 1 4567.0 1666.500000 2 6800.0 2287.053757 3 6810.0 2280.329732 4 6821.0 2186.267193 5 6820.0 898.332366 6 6820.0 8.158431 7 6820.0 4.118252 8 6820.0 0.400000 9 6820.0 0.000000 10 6820.0 0.000000  Now precision is better.
Question: I am using the HuggingFace Transformers package to access pretrained models. As my use case needs functionality for both English and Arabic, I am using the bert-base-multilingual-cased pretrained model. I need to be able to compare the similarity of sentences using something such as cosine similarity. To use this, I first need to get an embedding vector for each sentence, and can then compute the cosine similarity. Firstly, what is the best way to extratc the semantic embedding from the BERT model? Would taking the last hidden state of the model after being fed the sentence suffice? import torch from transformers import BertModel, BertTokenizer model_class = BertModel tokenizer_class = BertTokenizer pretrained_weights = 'bert-base-multilingual-cased' tokenizer = tokenizer_class.from_pretrained(pretrained_weights) model = model_class.from_pretrained(pretrained_weights) sentence = 'this is a test sentence' input_ids = torch.tensor([tokenizer.encode(sentence, add_special_tokens=True)]) with torch.no_grad(): output_tuple = model(input_ids) last_hidden_states = output_tuple[0] print(last_hidden_states.size(), last_hidden_states)  Secondly, if this is a sufficient way to get embeddings from my sentence, I now have another problem where the embedding vectors have different lengths depending on the length of the original sentence. The shapes output are [1, n, vocab_size], where n can have any value.  In order to compute two vectors' cosine similarity, they need to be the same length. How can I do this here? Could something as naive as first summing across axis=1 still work? What other options do I have? 

Answer: You can use the [CLS] token as a representation for the entire sequence. This token is typically prepended to your sentence during the preprocessing step. This token that is typically used for classification tasks (see figure 2 and paragraph 3.2 in the BERT paper). It is the very first token of the embedding. Alternatively you can take the average vector of the sequence (like you say over the first(?) axis), which can yield better results according to the huggingface documentation (3rd tip). Note that BERT was not designed for sentence similarity using the cosine distance, though in my experience it does yield decent results.
Question: I have a pandas dataframe which contains thousands of rows, and a few columns. I am getting an error when trying to convert it to a json file.  This is the code to convert: sessionAttendance.to_json('SessionAttendance.json')  This is the error I'm getting: OverflowError: Maximum recursion level reached _id wondeID session updatedAt 0 123456789101112131415161 AA1234567891 AM 2019-06-21 08:05:50.845 1 123456789101112131415162 AA1234567892 AM 2019-06-21 08:05:50.845 2 123456789101112131415163 AA1234567893 AM 2019-06-21 08:05:50.845 3 123456789101112131415164 AA1234567894 AM 2019-06-21 08:05:50.845 [234195 rows x 4 columns] 

Answer: It seems to be related to the way Mongo formats its _id fields which are not correctly processed by the json module. A workaround is to set default_handler=str to force the json formatter to use a string representation for any unwanted type: sessionAttendance.to_json('SessionAttendance.json', default_handler=str)  Disclaimer: credit should be given to that other SO post
Question: I'm new to OpenCV. Some weird things happened when I drew a circle. It didn't work when I tried to pass c2 to the circle function, but it worked well when I pass c1 to the color argument. But c1 == c2. Here is my code: import cv2 import numpy as np canvas = np.zeros((300, 300, 3), dtype='uint8') for _ in range(1): r = np.random.randint(0, 200) center = np.random.randint(0, 300, size=(2, )) color = np.random.randint(0, 255, size=(3, )) c1 = tuple(color.tolist()) c2 = tuple(color) print('c1 == c2 : {} '.format(c1 == c2)) cv2.circle(canvas, tuple(center), r, c2, thickness=-1) cv2.imshow('Canvas', canvas) cv2.waitKey(0)  When I use c2, the console printed:  TypeError: Scalar value for argument 'color' is not numeric  but why it happened when c1 == c2? Thanks.

Answer:  Convert data type int64 to int. ndarray.tolist() : data items are converted to the nearest compatible builtin Python type, via the item function.  Ex. import cv2 import numpy as np canvas = np.zeros((300, 300, 3), dtype='uint8') for _ in range(1): r = np.random.randint(0, 200) center = np.random.randint(0, 300, size=(2, )) color = np.random.randint(0, 255, size=(3, )) #convert data types int64 to int color = ( int (color [ 0 ]), int (color [ 1 ]), int (color [ 2 ])) cv2.circle(canvas, tuple(center), r, tuple (color), thickness=-1) cv2.imshow('Canvas', canvas) cv2.waitKey(0)
Question: How can I plot in Python a Confusion Matrix similar do the one shown here for already given values of the Confusion Matrix? In the code they use the method sklearn.metrics.plot_confusion_matrix which computes the Confusion Matrix based on the ground truth and the predictions. But in my case, I already have calculated my Confusion Matrix. So for example, my Confusion Matrix is (values in percentages): [[0.612, 0.388] [0.228, 0.772]] 

Answer: If you check the source for sklearn.metrics.plot_confusion_matrix, you can see how the data is processed to create the plot. Then you can reuse the constructor ConfusionMatrixDisplay and plot your own confusion matrix. import matplotlib.pyplot as plt from sklearn.metrics import ConfusionMatrixDisplay cm = [0.612, 0.388, 0.228, 0.772] # your confusion matrix ls = [0, 1] # your y labels disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=ls) disp.plot(include_values=include_values, cmap=cmap, ax=ax, xticks_rotation=xticks_rotation) plt.show()
Question: I am using torch summary  from torchsummary import summary  I want to pass more than one argument when printing the model summary, but the examples mentioned here: Model summary in pytorch taken only one argument. for e.g.: model = Network().to(device) summary(model,(1,28,28))  The reason is that the forward function takes two arguments as input, e.g.: def forward(self, img1, img2):  How do I pass two arguments here?

Answer: You can use the example given here: pytorch summary multiple inputs summary(model, [(1, 16, 16), (1, 28, 28)])
Question: I'm trying to rearrange columns in a DataFrame, by putting a few columns first, and then all the others after. With R's dplyr, this would look like: library(dplyr) df = tibble(col1 = c("a", "b", "c"), id = c(1, 2, 3), col2 = c(2, 4, 6), date = c("1 Feb", "2 Feb", "3 Feb")) df2 = select(df, id, date, everything())  Easy. With Python's pandas, here's what I've tried: import pandas as pd df = pd.DataFrame({ "col1": ["a", "b", "c"], "id": [1, 2, 3], "col2": [2, 4, 6], "date": ["1 Feb", "2 Feb", "3 Feb"] }) # using sets cols = df.columns.tolist() cols_1st = {"id", "date"} cols = set(cols) - cols_1st cols = list(cols_1st) + list(cols) # wrong column order df2 = df[cols] # using lists cols = df.columns.tolist() cols_1st = ["id", "date"] cols = [c for c in cols if c not in cols_1st] cols = cols_1st + cols # right column order, but is there a better way? df3 = df[cols]  The pandas way is more tedious, but I'm fairly new to this. Is there a better way?

Answer: You can use df.drop: >>> df = pd.DataFrame({ "col1": ["a", "b", "c"], "id": [1, 2, 3], "col2": [2, 4, 6], "date": ["1 Feb", "2 Feb", "3 Feb"] }) >>> df col1 id col2 date 0 a 1 2 1 Feb 1 b 2 4 2 Feb 2 c 3 6 3 Feb >>> cols_1st = ["id", "date"] >>> df[cols_1st + list(df.drop(cols_1st, 1))] id date col1 col2 0 1 1 Feb a 2 1 2 2 Feb b 4 2 3 3 Feb c 6
Question: Trying to create a new workitem in VSTS via Python API access and I cant find anywhere in the documents on how to create a new workitem in Python. I'm sure it's fairly simple but I can't seem to find it in the documentation. https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/work%20items/create?view=azure-devops-rest-5.1

Answer: Please kindly refer this official Azure DevOps Python API doc.  It contains Python APIs for interacting with and managing Azure DevOps. These APIs power the Azure DevOps Extension for Azure CLI. To learn more about the Azure DevOps Extension for Azure CLI, visit the  Microsoft/azure-devops-cli-extension repo.  Here is some example code for creating work item in python.
Question: The CuDNNGRU in TensorFlow 1.0 is really fast. But when I shifted to TensorFlow 2.0 i am unable to find CuDNNGRU. Simple GRU is really slow in TensorFlow 2.0. Is there any way to use CuDNNGRU in TensorFlow 2.0?

Answer: The importable implementations have been deprecated - instead, LSTM and GRU will default to CuDNNLSTM and CuDNNGRU if all conditions are met:  activation = 'tanh' recurrent_activation = 'sigmoid' recurrent_dropout = 0 unroll = False use_bias = True Inputs, if masked, are strictly right-padded reset_after = True (GRU only)   Also ensure TensorFlow uses GPU: import tensorflow as tf from tensorflow.python.client import device_lib print(device_lib.list_local_devices()) print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))   Update: there appears to be a problem w/ TF 2.0.0 when running on Colab in getting CuDNN to work; try !pip install tensorflow==2.1.0 instead.