eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a FastAPI app that is working as expected when running locally, however, I get an 'Internal Server Error' when I try to run in a Docker container. Here's the code for my app: from fastapi import FastAPI from pydantic import BaseModel import pandas as pd from fbprophet import Prophet class Data(BaseModel): length: int ds: list y: list model: str changepoint: float = 0.5 daily: bool = False weekly: bool = False annual: bool = False upper: float = None lower: float = 0.0 national_holidays: str = None app = FastAPI() @app.post("/predict/") async def create_item(data: Data): # Create df from base model df = pd.DataFrame(list(zip(data.ds, data.y)), columns =['ds', 'y']) # Add the cap and floor to df for logistic model if data.model == "logistic": df['y'] = 10 - df['y'] df['cap'] = data.upper df['floor'] = data.lower # make basic prediction m = Prophet(growth=data.model, changepoint_prior_scale=data.changepoint, weekly_seasonality=data.weekly, daily_seasonality=data.daily, yearly_seasonality=data.annual ) # Add national holidays if data.national_holidays is not None: m.add_country_holidays(country_name=data.national_holidays) # Fit data frame m.fit(df) # Create data frame for future future = m.make_future_dataframe(periods=data.length) # Add the cap and floor to future for logistic model if data.model == "logistic": future['cap'] = 6 future['floor'] = 1.5 # forecast forecast = m.predict(future) # Print values print(list(forecast[['ds']].values)) # Return results # {'ds': forecast[['ds']], 'yhat': forecast[['yhat']], 'yhat_lower': forecast[['yhat_lower']], 'yhat_upper': forecast[['yhat_upper']] } return [forecast[['ds']], forecast[['yhat']], forecast[['yhat_lower']], forecast[['yhat_upper']]]  Which is working locally with uvicorn main:app, but not when I build using this Dockerfile: FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7 COPY ./app /app RUN pip install -r requirements.txt  and start with docker run -d --name mycontainer -p 8000:80 myimage  I'm seeing Internal Server Error in Postman. Is there something wrong with my dockerfile or docker commands? Or else how do I debug this?

Answer: Run the docker without the -d parameter and you'll get more clues about it. If I were to guess, I might say that you're missing some python requirement.
Question: I have a data base file .db in SQLite3 format and I was attempting to open it to look at the data inside it. Below is my attempt to code using python.  import sqlite3 # Create a SQL connection to our SQLite database con = sqlite3.connect(dbfile) cur = con.cursor() # The result of a "cursor.execute" can be iterated over by row for row in cur.execute("SELECT * FROM "): print(row) # Be sure to close the connection con.close()  For the line ("SELECT * FROM ") , I understand that you have to put in the header of the table after the word "FROM", however, since I can't even open up the file in the first place, I have no idea what header to put. Hence how can I code such that I can open up the data base file to read its contents?

Answer: So, you analyzed it all right. After the FROM you have to put in the tablenames. But you can find them out like this: SELECT name FROM sqlite_master WHERE type = 'table'  In code this looks like this: # loading in modules import sqlite3 # creating file path dbfile = '/home/niklas/Desktop/Stuff/StockData-IBM.db' # Create a SQL connection to our SQLite database con = sqlite3.connect(dbfile) # creating cursor cur = con.cursor() # reading all table names table_list = [a for a in cur.execute("SELECT name FROM sqlite_master WHERE type = 'table'")] # here is you table list print(table_list) # Be sure to close the connection con.close()  That worked for me very good. The reading of the data you have done already right just paste in the tablenames.
Question: Checking out the zoneinfo module in Python 3.9, I was wondering if it also offers a convenient option to retrieve the local time zone (OS setting) on Windows. On GNU/Linux, you can do from datetime import datetime from zoneinfo import ZoneInfo naive = datetime(2020, 6, 11, 12) aware = naive.replace(tzinfo=ZoneInfo('localtime'))  but on Windows, that throws  ZoneInfoNotFoundError: 'No time zone found with key localtime'  so would I still have to use a third-party library? e.g. import time import dateutil tzloc = dateutil.tz.gettz(time.tzname[time.daylight]) aware = naive.replace(tzinfo=tzloc)  Since time.tzname[time.daylight] returns a localized name (German in my case, e.g. 'Mitteleuropäische Sommerzeit'), this doesn't work either: aware = naive.replace(tzinfo=ZoneInfo(tzloc))  Any thoughts?  p.s. to try this on Python < 3.9, use backports (see also this answer): pip install backports.zoneinfo pip install tzdata # needed on Windows 

Answer: You don't need to use zoneinfo to use the system local time zone. You can simply pass None (or omit) the time zone when calling datetime.astimezone.  From the docs:  If called without arguments (or with tz=None) the system local timezone is assumed. The .tzinfo attribute of the converted datetime instance will be set to an instance of timezone with the zone name and offset obtained from the OS.  Thus: from datetime import datetime naive = datetime(2020, 6, 11, 12) aware = naive.astimezone()
Question: I am facing this issue while using Mask_RCNN to train on my custom dataset with multiple classes. This error occurs when I start training. This is what I get: /home/parth/anaconda3/envs/compVision/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning. order = _validate_interpolation_order(image.dtype, order) I keep getting this for like a hundred times and then the kernel dies. Please Help!!

Answer: Maybe you can try the skimage version 0.16.2。when I use the version 0.17.2， I faced the same issue.Good luck!Idont know why.
Question: I do understand that higher-order functions are functions that take functions as parameters or return functions. I also know that decorators are functions that add some functionality to other functions. What are they exactly? Are they the functions that are passed in as parameters or are they the higher-order functions themselves? Note: If you give an example, use python.

Answer: A higher order function is a function that takes a function as an argument OR* returns a function. A decorator in Python is (typically) an example of a higher-order function, but there are decorators that aren't (class decorators**, and decorators that aren't functions), and there are higher-order functions that aren't decorators, for example those that take two required arguments that are functions. Not decorator, not higher-order function: def hello(who): print("Hello", who)  Not decorator, but higher-order function: def compose(f, g): def wrapper(*args, **kwargs): return g(f(*args, **kwargs)) return wrapper  Decorator, not higher-order function: def classdeco(cls): cls.__repr__ = lambda self: "WAT" return cls # Usage: @classdeco class Foo: pass  Decorator, higher-order function: def log_calls(fn): def wrapper(*args, **kwargs): print("Calling", fn.__name__) return fn(*args, **kwargs) return wrapper   * Not XOR ** Whether or not you consider class decorators to be higher-order functions, because classes are callable etc. is up for debate, I guess..
Question: I have three lists as follows. mylist = [[5274919, ["my cat", "little dog", "fish", "rat"]], [5274920, ["my cat", "parrot", "little dog"]], [5274991, ["little dog", "fish", "duck"]]] myconcepts = ["my cat", "little dog"] hatedconcepts = ["rat", "parrot"]  For each concept in myconcepts, I want to get the count every other concept connected to it using mylist. Then remove the hatedconcepts from it. So, my output should looks like as follows. {"my cat": [("my cat", 2), ("little dog", 2), ("fish", 1)], "little dog": [("little dog", 3), ("my cat", 2), ("fish", 2), ("duck", 1)]}  I was using this code to do it. import collections myoutput = [] for concept in myconcepts: mykeywords = [] for item in mylist: if concept in item[1]: for mykeyword in item[1]: if mykeyword in hatedconcepts: pass else: mykeywords.append(mykeyword) if len(mykeywords) > 0: sorted_keywords = collections.Counter(mykeywords).most_common() myoutput.append(tuple((concept, sorted_keywords))) print(myoutput)  The output of the code is: [('my cat', [('my cat', 2), ('little dog', 2), ('fish', 1)]), ('little dog', [('little dog', 3), ('my cat', 2), ('fish', 2), ('duck', 1)])]  However now I have a huge mylist with a size of 3GB and nearly 9000 myconcepts. The hatedconcepts count is only 20. It looks like it takes about two weeks to run using my current code. The main reason for this could be that my current program is O^3 which is not very efficient. Therefore, I am looking for ways to make my current program more efficient. I am even fine with pythonic solutions that even take 5-6 days to run. Please let me know your thoughts. I have added a portion of mylist in: https://drive.google.com/file/d/1M3EhIRwwKwD3Kv4zDsmXaH1D73tx0eF3/view?usp=sharing just to get some idea how it looks like. I am happy to provide more details if needed.

Answer: I have tried to make it fast, avoided some repeated loops. Please check if this speeds things up. from itertools import chain from collections import Counter, defaultdict database = defaultdict(set) output = {} # created a map for different concepts, so we only search the indices where a certain concept is for index, (_, concepts) in enumerate(mylist): for concept in concepts: database[concept].add(index) for concept in myconcepts: search_indices = database[concept] all_counts = Counter(chain.from_iterable(mylist[i][1] for i in search_indices)) for hc in hatedconcepts: if hc in all_counts: all_counts.pop(hc) output[concept] = sorted(all_counts.items(), key=lambda x: x[1], reverse=True)
Question: It's a pretty simple example import pandas df = pandas.DataFrame() value_to_be_set = {'1'} df.loc[0, 'col1'] = value_to_be_set df['col2'] = None df.loc[0, 'col2'] = value_to_be_set print(df.head())  output  col1 col2 0 1 {1}  Why is the datatype different for both columns? Python 3.7.3 pandas version: 0.23.4

Answer: In first assignment, you create a num_column from a set, said differently from an iterable. You ask for 1 single element and provide an iterable of size one, so you affect the content of the set to the single cell. You can try to use a set of 2 values to see that it would raise an error. In second assignment, you update a cell in an existing column. Pandas has no reason to unpack anything here, and it affects the set to the cell. To be honest, this explains what happens, but is not a justification for the rationale behind the different behaviours...
Question: The pandas.DataFrame.to_numpy method has a copy argument with the following documentation:  copy : bool, default False Whether to ensure that the returned value is a not a view on another array. Note that copy=False does not ensure that to_numpy() is no-copy. Rather, copy=True ensure that a copy is made, even if not strictly necessary.  Playing around a bit, it seems like calling to_numpy on data that is both adjacent in memory and not of mixed types, keeps a view. But how do I check whether the resulting numpy array shares the memory with the data frame it was created from, without changing data? Example of memory sharing: import pandas as pd import numpy as np # some data frame that I expect not to be copied frame = pd.DataFrame(np.arange(144).reshape(12,12)) array = frame.to_numpy() array[:] = 0 print(frame) # Prints: # 0 1 2 3 4 5 6 7 8 9 10 11 # 0 0 0 0 0 0 0 0 0 0 0 0 0 # 1 0 0 0 0 0 0 0 0 0 0 0 0 # 2 0 0 0 0 0 0 0 0 0 0 0 0 # 3 0 0 0 0 0 0 0 0 0 0 0 0 # 4 0 0 0 0 0 0 0 0 0 0 0 0 # 5 0 0 0 0 0 0 0 0 0 0 0 0 # 6 0 0 0 0 0 0 0 0 0 0 0 0 # 7 0 0 0 0 0 0 0 0 0 0 0 0 # 8 0 0 0 0 0 0 0 0 0 0 0 0 # 9 0 0 0 0 0 0 0 0 0 0 0 0 # 10 0 0 0 0 0 0 0 0 0 0 0 0 # 11 0 0 0 0 0 0 0 0 0 0 0 0  Example not sharing memory: import pandas as pd import numpy as np # some data frame that I expect to be copied types = [int, str, float] frame = pd.DataFrame({ i: [types[i%len(types)](value) for value in col] for i, col in enumerate(np.arange(144).reshape(12,12).T) }) array = frame.to_numpy() array[:] = 0 print(frame) # Prints: # 0 1 2 3 4 5 6 7 8 9 10 11 # 0 0 12 24.0 36 48 60.0 72 84 96.0 108 120 132.0 # 1 1 13 25.0 37 49 61.0 73 85 97.0 109 121 133.0 # 2 2 14 26.0 38 50 62.0 74 86 98.0 110 122 134.0 # 3 3 15 27.0 39 51 63.0 75 87 99.0 111 123 135.0 # 4 4 16 28.0 40 52 64.0 76 88 100.0 112 124 136.0 # 5 5 17 29.0 41 53 65.0 77 89 101.0 113 125 137.0 # 6 6 18 30.0 42 54 66.0 78 90 102.0 114 126 138.0 # 7 7 19 31.0 43 55 67.0 79 91 103.0 115 127 139.0 # 8 8 20 32.0 44 56 68.0 80 92 104.0 116 128 140.0 # 9 9 21 33.0 45 57 69.0 81 93 105.0 117 129 141.0 # 10 10 22 34.0 46 58 70.0 82 94 106.0 118 130 142.0 # 11 11 23 35.0 47 59 71.0 83 95 107.0 119 131 143.0 

Answer: There is numpy.shares_memory you can use: # Your first example print(np.shares_memory(array, frame)) # True, they are sharing memory # Your second example print(np.shares_memory(array2, frame2)) # False, they are not sharing memory  There is also numpy.may_share_memory, which is faster but can only be used for making sure things do not share memory (because it only checks whether the bounds overlap), so strictly speaking does not answer the question. Read this for the differences. Take care using these numpy functions with pandas data-structures: np.shares_memory(frame, frame) returns True for the first example, but False for the second, probably because the __array__ method of the data frame in the second example creates a copy behind the scenes.
Question: I'm trying to make a simple image classifier using PyTorch. This is how I load the data into a dataset and dataLoader: batch_size = 64 validation_split = 0.2 data_dir = PROJECT_PATH+"/categorized_products" transform = transforms.Compose([transforms.Grayscale(), CustomToTensor()]) dataset = ImageFolder(data_dir, transform=transform) indices = list(range(len(dataset))) train_indices = indices[:int(len(indices)*0.8)] test_indices = indices[int(len(indices)*0.8):] train_sampler = SubsetRandomSampler(train_indices) test_sampler = SubsetRandomSampler(test_indices) train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler, num_workers=16) test_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=test_sampler, num_workers=16)  I want to print out the number of images in each class in training and test data separately, something like this: In train data:  shoes: 20 shirts: 14  In test data:  shoes: 4 shirts: 3  I tried this: from collections import Counter print(dict(Counter(sample_tup[1] for sample_tup in dataset.imgs)))  but I got this error: AttributeError: 'MyDataset' object has no attribute 'img' 

Answer: You need to use .targets to access the labels of data i.e. print(dict(Counter(dataset.targets)))  It'll print something like this (e.g. in MNIST dataset): {5: 5421, 0: 5923, 4: 5842, 1: 6742, 9: 5949, 2: 5958, 3: 6131, 6: 5918, 7: 6265, 8: 5851}  Also, you can use .classes or .class_to_idx to get mapping of label id to classes: print(dataset.class_to_idx) {'0 - zero': 0, '1 - one': 1, '2 - two': 2, '3 - three': 3, '4 - four': 4, '5 - five': 5, '6 - six': 6, '7 - seven': 7, '8 - eight': 8, '9 - nine': 9}   Edit: Method 1 From the comments, in order to get class distribution of training and testing set separately, you can simply iterate over subset as below: train_size = int(0.8 * len(dataset)) test_size = len(dataset) - train_size train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, test_size]) # labels in training set train_classes = [label for _, label in train_dataset] Counter(train_classes) Counter({0: 4757, 1: 5363, 2: 4782, 3: 4874, 4: 4678, 5: 4321, 6: 4747, 7: 5024, 8: 4684, 9: 4770})   Edit (2): Method 2 Since you've a large dataset, and as you said it takes considerable time to iterate over all training set, there is another way: You can use .indices of subset, which referes to indices in the original dataset selected for subset. i.e.  train_classes = [dataset.targets[i] for i in train_dataset.indices] Counter(train_classes) # if doesn' work: Counter(i.item() for i in train_classes)
Question: So I'm trying to prepare a message with Python that takes a Timestamp, but I'm having trouble converting a datetime to a protobuf Timestamp. Here's what I've tried so far: from google.protobuf.timestamp_pb2 import Timestamp import datetime now = datetime.datetime.now() timestamp = Timestamp() timestamp.FromDatetime(now)  However, I'm getting an error AttributeError: 'Timestamp' object attribute 'seconds' is read-only How can I create a Timestamp from a datetime?

Answer: This code is working fine on my machine from google.protobuf.timestamp_pb2 import Timestamp import datetime now = datetime.datetime.now() timestamp = Timestamp() timestamp.FromDatetime(now)  Output: seconds: 1591859232 nanos: 803377000
Question: I use pybind11 as a wrapper of my C++ code into a python library. It happens that there are arguments that I can't provide or sometimes I want to do a conversion/initialization that I know in the C++ side. It could be because the class is not known in python, for instance. How could that be done? The only "solution" I see so far would be to create an inherited proxy class in C++. Example: I want to define/bind a python class A: class A: def __init__(self, B b): ...  With a C++ equivalent class: class A { A(C c, D d); }  Is there some kind of lambda or an equivalent that I could create for the pybind11::init<>?

Answer: pybind11 lets you bind factory functions as init methods. So you would have to provide a function in c++ that took a B and return an A and then you could bind that as an init method for A. An example from the pybind11 docs class Example { private: Example(int); // private constructor public: // Factory function: static Example create(int a) { return Example(a); } }; py::class_<Example>(m, "Example") .def(py::init(&Example::create));  You should be able to bind in a free function as well (not just a static function), if you do not want to (or can't) change class A in c++. So it could look something like this (changed to return a unique_ptr, which pybind can just take ownership of vs a raw instance. But either should work) std::unique_ptr<A> createA(const B& arg) { // returns an instance of A that you made using B } py::class_<A>(m, "A") .def(py::init(&createA));  You obviously then have to also provide a binding for B in python. Docs are here and include even more examples, including how to do an init lambda as well: https://pybind11.readthedocs.io/en/stable/advanced/classes.html#custom-constructors
Question: I just want to install pytorch, I ran this in the terminal: pip install torch  And it displays: Collecting torch Killed  What is the problem?

Answer: It says your your free ram is not enough to install the package, but there is a method that you can still use it. pip install torch --no-cache-dir
Question: Is there a straight-forward approach to generate a Pydantic model from a dictionary? Here is a sample of the data I have. { 'id': '424c015f-7170-4ac5-8f59-096b83fe5f5806082020', 'contacts': [{ 'displayName': 'Norma Fisher', 'id': '544aa395-0e63-4f9a-8cd4-767b3040146d' }], 'startTime': '2020-06-08T09:38:00+00:00' }  Expecting a model similar to ... class NewModel(BaseModel): id: str contacts: list startTime: str 

Answer: In Pydantic 2, you can use MyModel.model_validate(my_dict) to generate a model from a dictionary. According to the documentation –  this is very similar to the __init__ method of the model, except it takes a dict rather than keyword arguments.  If you're Pydantic 1, the method is parse_obj instead.
Question: I'm try to adding watermark to every pages of my PDF file.My PDF files have 58 pages but my output file has get only last page in my PDF file.   This's my code: from PyPDF2 import PdfFileReader, PdfFileWriter watermark_pdf = PdfFileReader("watermark.pdf") watermark_page = watermark_pdf.getPage(0) reader = PdfFileReader("original_document.pdf") for page in reader.pages: page.mergePage(watermark_page) output = PdfFileWriter() output.addPage(page) with open("watermarked_document.pdf", "wb") as fp: output.write(fp)  Please tell me how to add watermark all pages.

Answer: You're rewriting your "merged" file for each page. Try something like from PyPDF2 import PdfFileMerger, PdfFileReader, PdfFileWriter pdf_file = "C:/Users/11359023/Desktop/deepfake_vee.pdf" watermark = "C:/Users/11359023/Desktop/simple.pdf" merged = "C:/Users/11359023/Desktop/merged.pdf" with open(pdf_file, "rb") as input_file, open(watermark, "rb") as watermark_file: input_pdf = PdfFileReader(input_file) watermark_pdf = PdfFileReader(watermark_file) watermark_page = watermark_pdf.getPage(0) output = PdfFileWriter() for i in range(input_pdf.getNumPages()): pdf_page = input_pdf.getPage(i) pdf_page.merge_page(watermark_page) output.addPage(pdf_page) with open(merged, "wb") as merged_file: output.write(merged_file)  instead. Edit:- mergePage() has been Deprecated and removed in PyPDF2 3.0.0. use merge_page() instead.
Question: Using the 'tips' dataset as a toy model, I generate the following plot:  import seaborn as sns import matplotlib.pyplot as plt tips = sns.load_dataset("tips") cmap = sns.cubehelix_palette(dark=.3, light=.8, as_cmap=True) g = sns.scatterplot(x="total_bill", y="sex", hue="smoker", size = 'tip',sizes=(320, 600), data=tips) plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., fontsize=13) plt.show(g)  This image is exactly what I need. However, I want to remove the size = 'tip' from the legend and only keep the smoker. Essentially, remove those black circles labeled 0.0 to 12.0. How do I ensure my legend has only one variable of my choosing? 

Answer: I was able to find a fix by indexing the labels in the legend. import seaborn as sns import matplotlib.pyplot as plt tips = sns.load_dataset("tips") cmap = sns.cubehelix_palette(dark=.3, light=.8, as_cmap=True) ax = sns.scatterplot(x="total_bill", y="sex", hue="smoker", size='tip', sizes=(320, 600), data=tips) # extract the existing handles and labels h, l = ax.get_legend_handles_labels() # slice the appropriate section of l and h to include in the legend ax.legend(h[0:3], l[0:3], bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., fontsize=13) plt.show()