eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Usually people use scikit-learn to train a model this way: from sklearn.ensemble import GradientBoostingClassifier as gbc clf = gbc() clf.fit(X_train, y_train) predicted = clf.predict(X_test)  It works fine as long as users' memory is large enough to accommodate the entire dataset. The dilemma for me is exactly this--the dataset is too big for my memory. My current solution is to enlarge the virtual memory of my machine and I have already made the system extremely slow by having too much virtual memory--so I start to think whether or not is it possible to feed the fit() method with samples in batches like this (and the answre is no, please keep reading and stop reminding me that the answer is no): clf = gbc() for i in range(X_train.shape[0]): clf.fit(X_train[i], y_train[i])  so that I can read the training set from hard drive only when needed. I read the sklearn's manual and it seems to me that it does not support this: Calling fit() more than once will overwrite what was learned by any previous fit()  So, is this possible?

Answer: After reading the section 6. Strategies to scale computationally: bigger data of the official manual mentioned by @StupidWolf in this post, I am aware that this question is more to this than meets the eye. The real difficulty is about the design of a lot of models. Take Random Forest as an example, one of the most important techniques used to improve its performance compared with the simpler Decision Tree is the application of bagging, which means that the algorithm has to pick some random samples from the entire dataset to construct several weak learners as the basis of the Random Forest. It means that feeding the model with one sample after another won't work with this design. Although it is still possible for scikit-learn to define an interface for end-users to implement so that scikit-learn can pick a random sample by calling this interface and end-users will decide how their implementation of the interface is about to return the needed data by scanning the dataset on the hard drive, it becomes way more complicated than I initially thought and the performance gain may not be very significant given that the IO-heavy "full table scan" (in database's term) is frequently needed.
Question: I'm having trouble finding the solution to a fairly simple problem. I would like to alphabetically arrange certain columns of a pandas dataframe that has over 100 columns (i.e. so many that I don't want to list them manually). Example df: import pandas as pd subject = [1,1,1,1,1,1,2,2,2,2,2,2,3,3,3,4,4,4,4,4,4] timepoint = [1,2,3,4,5,6,1,2,3,4,5,6,1,2,4,1,2,3,4,5,6] c = [2,3,4,5,6,7,3,4,1,2,3,4,5,4,5,8,4,5,6,2,3] d = [2,3,4,5,6,7,3,4,1,2,3,4,5,4,5,8,4,5,6,2,3] a = [2,3,4,5,6,7,3,4,1,2,3,4,5,4,5,8,4,5,6,2,3] b = [2,3,4,5,6,7,3,4,1,2,3,4,5,4,5,8,4,5,6,2,3] df = pd.DataFrame({'subject':subject, 'timepoint':timepoint, 'c':c, 'd':d, 'a':a, 'b':b}) df.head() subject timepoint c d a b 0 1 1 2 2 2 2 1 1 2 3 3 3 3 2 1 3 4 4 4 4 3 1 4 5 5 5 5 4 1 5 6 6 6 6  How could I rearrange the column names to generate a df.head() that looks like this:  subject timepoint a b c d 0 1 1 2 2 2 2 1 1 2 3 3 3 3 2 1 3 4 4 4 4 3 1 4 5 5 5 5 4 1 5 6 6 6 6  i.e. keep the first two columns where they are and then alphabetically arrange the remaining column names. Thanks in advance.

Answer: You can split your your dataframe based on column names, using normal indexing operator [], sort alphabetically the other columns using sort_index(axis=1), and concat back together: >>> pd.concat([df[['subject','timepoint']], df[df.columns.difference(['subject', 'timepoint'])]\ .sort_index(axis=1)],ignore_index=False,axis=1) subject timepoint a b c d 0 1 1 2 2 2 2 1 1 2 3 3 3 3 2 1 3 4 4 4 4 3 1 4 5 5 5 5 4 1 5 6 6 6 6 5 1 6 7 7 7 7 6 2 1 3 3 3 3 7 2 2 4 4 4 4 8 2 3 1 1 1 1 9 2 4 2 2 2 2 10 2 5 3 3 3 3 11 2 6 4 4 4 4 12 3 1 5 5 5 5 13 3 2 4 4 4 4 14 3 4 5 5 5 5 15 4 1 8 8 8 8 16 4 2 4 4 4 4 17 4 3 5 5 5 5 18 4 4 6 6 6 6 19 4 5 2 2 2 2 20 4 6 3 3 3 3
Question: I need to connect to Snowflake using SQLAlchemy but the trick is, I need to authenticate using OAuth2. Snowflake documentation only describes connecting using username and password and this cannot be used in the solution I'm building. I can authenticate using Snowflake's python connector but I see no simple path how to glue it with SQLAlchemy. I'd like to know if there is a ready solution before I write a custom interface for this.

Answer: Use snowflake.connector.connect to create a PEP-249 Connection to the database - see documentation. Then use param creator of create_engine (docs) - it takes a callable that returns PEP-249 Connection. If you use it then URL param is ignored. Example code: def get_connection(): return snowflake.connector.connect( user="<username>", host="<hostname>", account="<account_identifier>", authenticator="oauth", token="<oauth_access_token>", warehouse="test_warehouse", database="test_db", schema="test_schema" ) engine = create_engine("snowflake://not@used/db", creator=get_connection)
Question: I have been using my python v3.9 in my virtual environment, it contains all the packages and scripts i have been using for so long. But, now with the release of python v 3.10 it installed itself globally though I wanted it to be installed in same venv I was using for python v3.9 So, if anyone could help me with how can I install python v3.10 in the same venv of my v3.9 . My IDE is PyCharm

Answer: Simply put all the dependencies of your python 3.9 (venv) in requirements.txt file pip freeze > requirements.txt  Create a new folder then move that file inside the newly created folder then execute the following code, it will create a new virtual environment with python 3.10 python -m venv newenv  activate the newly created environment by source newenv/bin/activate  then install the required dependencies by pip install -r requirements.txt  Note: If your OS do not have 'venv' module then simply install it by using pip install venv
Question: I would like to retrieve information from Google Arts & Culture using BeautifulSoup. I have checked many of the stackoverflow posts ([1], [2], [3], [4], [5]), and still couldn't retrieve the information. I would like each tile (picture)'s (li) information such as href, however, find_all and select one return empty list or None. Could you help me get the below href value of anchor tag of class "e0WtYb HpzMff PJLMUc" ? href="/entity/claude-monet/m01xnj?categoryId=artist" Below are what I had tried. import requests from bs4 import BeautifulSoup url = 'https://artsandculture.google.com/category/artist?tab=time&date=1850' html = requests.get(url) soup = BeautifulSoup(html.text, 'html.parser') print(soup.find_all('li', class_='DuHQbc')) # [] print(soup.find_all('a', class_='PJLMUc')) # [] print(soup.find_all('a', class_='e0WtYb HpzMff PJLMUc')) # [] print(soup.select_one('#tab_time > div > div:nth-child(2) > div > ul > li:nth-child(2) > a')) # None for elem in soup.find_all('a', class_=['e0WtYb', 'HpzMff', 'PJLMUc'], href=True): print(elem) # others with class 'e0WtYb' ... # and then something like elem['href']  https://artsandculture.google.com/category/artist?tab=time&date=1850  Copied selector from Chrome  #tab_time > div > div:nth-child(2) > div > ul > li:nth-child(2) > a 

Answer: Unfortunately, the problem is not that you're using BeautifulSoup wrong. The webpage that you're requesting appears to be missing its content! I saved html.text to a file for inspection:  Why does this happen? Because the webpage actually loads its content using JavaScript. When you open the site in your browser, the browser executes the JavaScript, which adds all of the artist squares to the webpage. (You may even notice the brief moment during which the squares aren't there when you first load the site.) On the other hand, requests does NOT execute JavaScript—it just downloads the contents of the webpage and saves them to a string. What can you do about it? Unfortunately, this means that scraping the website will be really tough. In such cases, I would suggest looking for an alternative source of information or using an API provided by the website.
Question: I am quite new to python so I need some help. I would like to create a code that checks if a folder with a given name is created, if not, it creates it, if it exists, goes to it and in it checks if there is another folder with a given name, if not, it creates it. In my code I don't know how to go to folder that already exist or has just been created and repeat IF. import os MYDIR = ("test") CHECK_FOLDER = os.path.isdir(MYDIR) if not CHECK_FOLDER: os.makedirs(MYDIR) print("created folder : ", MYDIR) else: print(MYDIR, "folder already exists.") 

Answer: If you want to create a folder inside a second folder or check the existence of them you can use builtin os library: import os PATH = 'folder_1/folder_2' if not os.path.exists(PATH): os.makedirs(PATH)  os.makedirs() create all folders in the PATH if they does not exist.
Question: I have this issue with heatmap from seaborn. I don't know how, but seaborn.heatmap() refuses to take in dataframe, it instead show the mentioned error. Seaborn, matplotlib and pandas is up-to-date and I'm using python 3.10 on Visual Studio. The code is just a sample code from seaborn.heatmap itself: import pandas as pd import seaborn as sns import matplotlib.pyplot as plt flights = sns.load_dataset("flights") flights = flights.pivot("month", "year", "passengers") ax=sns.heatmap(flights) plt.show() 

Answer: Use Python 3.9 (or 3.8, 3.7, 3.6) as it seems like both pandas and plt are not quite ready to be used with Python 3.10:
Question: We have a number of Python Databricks jobs that all use the same underlying Wheel package to install their dependencies. Installing this Wheel package even with a node that has been idling in a Pool still takes 90 seconds. Some of these jobs are very long-running so we would like to use Jobs computer clusters for the lower cost in DBUs. Some of these jobs are much shorter-running (<10 seconds) where the 90 second install time seems more significant. We have been considering using a hot cluster (All-Purpose Compute) for these shorter jobs. We would like to avoid the extra cost of the All-Purpose Compute if possible. Reading the Databricks documentation suggests that the Idle instances in the Pool are reserved for us but not costing us DBUs. Is there a way for us to pre-install the required libraries on our Idle instances so that when a job comes through we are able to immediately start processing it? Is there an alternate approach that can fulfill a similar use case?

Answer: You can't install libraries directly into nodes from pool, because the actual code is executed in the Docker container corresponding to Databricks Runtime. There are several ways to speedup installation of the libraries:  Create your own Docker image with all necessary libraries pre-installed, and pre-load Databricks Runtime version and your Docker image - this part couldn't be done via UI, so you need to use REST API (see description of preloaded_docker_images attribute), databrick-cli, or Databricks Terraform provider. The main disadvantage of custom Docker images is that some functionality isn't available out of box, for example, arbitrary files in Repos, web terminal, etc. (don't remember full list) Put all necessary libraries and their dependencies onto DBFS and install them via cluster init script. It's very important that you collect binary dependencies, not packages only with the source code, so you won't need to compile them when installing. This could be done once:  for Python this could be done with pip download --prefer-binary lib1 lib2 ... for Java/Scala you can use mvn dependency:get -Dartifact=<maven_coordinates>, that will download dependencies into ~/.m2/repository folder, from which you can copy jars to DBFS and in init script use cp /dbfs/.../jars/* /databricks/jars/ command for R, it's slightly more complicated, but is also doable
Question: The size of my input images are 68 x 224 x 3 (HxWxC), and the first Conv2d layer is defined as conv1 = torch.nn.Conv2d(3, 16, stride=4, kernel_size=(9,9)). Why is the size of the output feature volume 16 x 15 x 54? I get that there are 16 filters, so there is a 16 in the front, but if I use [(W−K+2P)/S]+1 to calculate dimensions, the dimensions are not divisible. Can someone please explain?

Answer: The calculation of feature maps is [(W−K+2P)/S]+1 and here [] brackets means floor division. In your example padding is zero, so the calculation is [(68-9+2*0)/4]+1 ->[14.75]=14 -> [14.75]+1 = 15 and [(224-9+2*0)/4]+1 -> [53.75]=53 -> [53.75]+1 = 54. import torch conv1 = torch.nn.Conv2d(3, 16, stride=4, kernel_size=(9,9)) input = torch.rand(1, 3, 68, 224) print(conv1(input).shape) # torch.Size([1, 16, 15, 54])  You may see different formulas too calculate feature maps. In PyTorch:  In general, you may see this:  However the result of both cases are the same
Question: My question is about this kata on Codewars. The function takes two sorted lists with distinct elements as arguments. These lists might or might not have common items. The task is find the maximum path sum. While finding the sum, if there any common items you can choose to change your path to the other list. The given example is like this: list1 = [0, 2, 3, 7, 10, 12] list2 = [1, 5, 7, 8] 0->2->3->7->10->12 => 34 0->2->3->7->8 => 20 1->5->7->8 => 21 1->5->7->10->12 => 35 (maximum path)  I solved the kata but my code doesn't match the performance criteria so I get execution timed out. What can I do for it? Here is my solution: def max_sum_path(l1:list, l2:list): common_items = list(set(l1).intersection(l2)) if not common_items: return max(sum(l1), sum(l2)) common_items.sort() s = 0 new_start1 = 0 new_start2 = 0 s1 = 0 s2 = 0 for item in common_items: s1 = sum(itertools.islice(l1, new_start1, l1.index(item))) s2 = sum(itertools.islice(l2, new_start2, l2.index(item))) new_start1 = l1.index(item) new_start2 = l2.index(item) s += max(s1, s2) s1 = sum(itertools.islice(l1, new_start1, len(l1))) s2 = sum(itertools.islice(l2, new_start2, len(l2))) s += max(s1, s2) return s 

Answer: Your algorithm is actually fast, just your implementation is slow. The two things that make it take overall O(n²) time:  l1.index(item) always searches from the start of the list. Should be l1.index(item, new_start1). itertools.islice(l1, new_start1, ...) creates an iterator for l1 and iterates over the first new_start1 elements before it reaches the elements you want. So just use a normal list slice instead.  Then it's just O(n log n) for the sorting and O(n) for everything else. And the sorting's O(n log n) is fast, might easily take less time than the O(n) part for any allowed input and even larger ones. Here's the rewritten version, gets accepted in about 6 seconds, just like the solutions from the other answers. def max_sum_path(l1:list, l2:list): common_items = list(set(l1).intersection(l2)) if not common_items: return max(sum(l1), sum(l2)) common_items.sort() s = 0 new_start1 = 0 new_start2 = 0 s1 = 0 s2 = 0 for item in common_items: next_start1 = l1.index(item, new_start1) # changed next_start2 = l2.index(item, new_start2) # changed s1 = sum(l1[new_start1 : next_start1]) # changed s2 = sum(l2[new_start2 : next_start2]) # changed new_start1 = next_start1 # changed new_start2 = next_start2 # changed s += max(s1, s2) s1 = sum(l1[new_start1:]) # changed s2 = sum(l2[new_start2:]) # changed s += max(s1, s2) return s  Or you could use iterators instead of indexes. Here's your solution rewritten to do that, also gets accepted in about 6 seconds: def max_sum_path(l1:list, l2:list): common_items = sorted(set(l1) & set(l2)) s = 0 it1 = iter(l1) it2 = iter(l2) for item in common_items: s1 = sum(iter(it1.__next__, item)) s2 = sum(iter(it2.__next__, item)) s += max(s1, s2) + item s1 = sum(it1) s2 = sum(it2) s += max(s1, s2) return s  I'd combine the last four lines into one, just left it like you had so it's easier to compare.
Question: When iterating through rows in a dataframe in Pandas, is there a difference in performance between using: for index in df.index: ....  And: for index, row in df.iterrows(): ....  ? Which one should be preferred?

Answer: When we doing for loop , look up index get the data require additional loc for index in df.index: value = df.loc['index','col']  When we do df.iterrows for index, row in df.iterrows(): value = row['col']  Since you already with pandas , both of them are not recommended. Unless you need certain function and cannot be vectorized. However, IMO, I preferred df.index
Question: We have deployed a model prediction service in production that is using FastAPI and unfortunately, some of the requests are failing due to a 10s timeout. In terms of concurrent requests, we typically only load about 2/3 requests per second, so I wouldn't think that would be too much strain on FastAPI. The first thing we tried to do is isolate the FastAPI framework from the model itself, and when we performed some tracing, we noticed that a lot of time (6 seconds) was spent on this segment: starlette.exceptions:ExceptionMiddleware.__call__. The gunicorn configuration we are using didn't seem to help either: """gunicorn server configuration.""" import os ​ threads = 2 workers = 4 timeout = 60 keepalive = 1800 graceful_timeout = 1800 bind = f":{os.environ.get('PORT', '80')}" worker_class = "uvicorn.workers.UvicornWorker"  Would really appreciate some guidance on what the above segment implies and what is causing timeout issues for some requests under a not too strenuous load.  

Answer:  guidance on what the above segment implies  here you have the official gunicorn config file with lot of explainations included. since you use gunicorn to manager uvicorn workers, forcing your timeout to 60 sec should work just fine for lnog running tasks (you should think about using a asynchronous task queue or job queue like celery) but what is returning your route ? first thing would be to see the error thrown by your api  starlette.exceptions:ExceptionMiddleware.call  Since you have expanded the list you can see that what take the most time (as expected) is not fastapi nor starlette but you function in app.api.routes.predictions.  so I wouldn't think that would be too much strain on FastAPI  it is not too much strain of fastapi since it is not involved in your request treatment. Remember that fastapi is "just" a framework so when your function take time it's your function/development that's at fault. here it can be one or a combinaison of thoses things that cause long runing tasks:  sync route blocking I/O function or treatment in your route function prediction algo that take a lot of time (too much maybe) bad worker class configuration for your type of treatment  When you do AI or nlp stuff often if take a lot of treatment time, regarding the integration of such models in api you use a task queue like celery. If your api is not at fault and your route not returning an error, just that it take a lot of time you should take a look at implementing task queue's.
Question: Why do I keep on getting this error? I try other codes too, but once it uses the get_feature_names_out function it will pop out this error. Below is my code: from sklearn.datasets._twenty_newsgroups import fetch_20newsgroups from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.naive_bayes import MultinomialNB # fast to train and achieves a decent F-score from sklearn import metrics import numpy as np def show_top10(classifier, vectorizer, categories): feature_names = vectorizer.get_feature_names_out() for i, category in enumerate(categories): top10 = np.argsort(classifier.coef_[i])[-10:] print("%s: %s" % (category, " ".join(feature_names[top10]))) newsgroups_train = fetch_20newsgroups(subset='train') print(list(newsgroups_train.target_names)) cats = ['alt.atheism', 'sci.space', 'rec.sport.baseball', 'rec.sport.hockey'] newsgroups_train = fetch_20newsgroups(subset='train', categories=cats) print(list(newsgroups_train.target_names)) print(newsgroups_train.filenames.shape) vectorizer = TfidfVectorizer() vectors = vectorizer.fit_transform(newsgroups_train.data) print(vectors.shape) 

Answer: This is probably because you are using an older scikit-learn version than the one this code was written for. get_feature_names_out is a method of the class sklearn.feature_extraction.text.TfidfVectorizer since scikit-learn 1.0. Previously, there was a similar method called get_feature_names. So you should update your scikit-learn package, or use the old method (not recommended).
Question: Hi all I am having trouble understanding how to use the output of sklearn.calibration.CalibratedClassifierCV. I have calibrated my binary classifier using this method, and results are greatly improved. However I am not sure how to interpret the results. sklearn guide states that, after calibration,  the output of predict_proba method can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that among the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class.  Now I would like to reduce false positive by applying a cutoff at .6 for the model to predict label True. Without the calibration, I would have simply used my_model.predict_proba() > .6. However, it seems that after calibration the meaning of predict_proba has changed, so I am not sure if I can do that anymore. From a quick testing it seems that predict and predict_proba follow the same logic I would expect before calibration. The output of: pred = my_model.predict(valid_x) proba= my_model.predict_proba(valid_x) pd.DataFrame({"label": pred, "proba": proba[:,1]})  is the following:  Where everything that has a probability of above .5 gets to be classifed as True, and everything below .5 as False. Can you confirm that, after calibration, I can still use predict_proba to apply a different cutoff to identify my labels? 2 https://scikit-learn.org/stable/modules/calibration.html#calibration

Answer: For me, you can actually use predict_proba() after calibration to apply a different cutoff. What happens within class CalibratedClassifierCV (as you noticed) is effectively that the output of predict() is based on the output of predict_proba() (see here for reference), i.e. np.argmax(self.predict_proba(X), axis=1) == self.predict(X). On the other side, for the non-calibrated classifier that you're passing to CalibratedClassifierCV (depending on whether it is a probabilistic classifier or not) the above equality may or may not hold (e.g. it does not for an SVC() classifier - see here, for instance, for some other details on this).
Question: I'd like to install goole chrome on Python of Docker image. So, I need install libappindicator1. However when I build this Dockerfile, I got error on libappindicator1 Dockerfile FROM python:3.9 # Install manually all the missing libraries RUN apt-get update RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libcairo2 libcups2 libfontconfig1 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libxss1 fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils fonts-takao-* # Install Chrome RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb RUN dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install  Error message E: Unable to locate package libappindicator1  How can I install libappindicator1 on Python of Docker image?

Answer: I solved this problem by modifying the python image tag. python: 3.8 ->  python: 3.8-buster When I use python: 3.8-bullseye I got the same error. So this error seems to be related with Debian 10 (bullseye). Note: buster is Debian 9 This is the reason, why Debian 10 (bullseye) can not install libappindicator1  5.3.1. Noteworthy obsolete packages   The deprecated libappindicator libraries are no longer provided. As a result, the related packages libappindicator1, libappindicator3-1 and libappindicator-dev are no longer available. This is expected to cause dependency errors for third-party software that still depends on libappindicator to provide system tray and indicator support.