eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Few hour ago my setup in google colab for selenium worked fine. Now it stopped working all of a sudden. This is a sample: !pip install selenium !apt-get update !apt install chromium-chromedriver from selenium import webdriver chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage') driver = webdriver.Chrome('chromedriver', chrome_options=chrome_options)  I get the error: WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: 1  Any ideas on solving it?

Answer: This error message... WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: 1  ...implies that the chromedriver service unexpectedly exited. This is because of the of an issue induced as the colab system was updated from v18.04 to ubuntu v20.04 LTS recently. The main reason is, with Ubuntu v20.04 LTS google-colaboratory no longer distributes chromium-browser outside of a snap package.  Quick Fix   @mco-gh created a new notebook following @metrizable's guidance (details below) which is working perfect as of now: https://colab.research.google.com/drive/1cbEvuZOhkouYLda3RqiwtbM-o9hxGLyC    Solution As a solution you can install a compatible version of chromium-browser from the Debian buster repository using the following code block published by @metrizable in the discussion Issues when trying to use Chromedriver in Colab %%shell # Ubuntu no longer distributes chromium-browser outside of snap # # Proposed solution: https://askubuntu.com/questions/1204571/how-to-install-chromium-without-snap # Add debian buster cat > /etc/apt/sources.list.d/debian.list <<'EOF' deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster.gpg] http://deb.debian.org/debian buster main deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster-updates.gpg] http://deb.debian.org/debian buster-updates main deb [arch=amd64 signed-by=/usr/share/keyrings/debian-security-buster.gpg] http://deb.debian.org/debian-security buster/updates main EOF # Add keys apt-key adv --keyserver keyserver.ubuntu.com --recv-keys DCC9EFBF77E11517 apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138 apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 112695A0E562B32A apt-key export 77E11517 | gpg --dearmour -o /usr/share/keyrings/debian-buster.gpg apt-key export 22F3D138 | gpg --dearmour -o /usr/share/keyrings/debian-buster-updates.gpg apt-key export E562B32A | gpg --dearmour -o /usr/share/keyrings/debian-security-buster.gpg # Prefer debian repo for chromium* packages only # Note the double-blank lines between entries cat > /etc/apt/preferences.d/chromium.pref << 'EOF' Package: * Pin: release a=eoan Pin-Priority: 500 Package: * Pin: origin "deb.debian.org" Pin-Priority: 300 Package: chromium* Pin: origin "deb.debian.org" Pin-Priority: 700 EOF # Install chromium and chromium-driver apt-get update apt-get install chromium chromium-driver
Question: Given a ParquetFile object (docs) I am able to retrieve data at row group / column chunk level either with read_row_group or with the metadata attribute: from pyarrow import fs from pyarrow.parquet import ParquetFile s3 = fs.S3FileSystem(region='us-east-2') path = 'voltrondata-labs-datasets/nyc-taxi/year=2009/month=1/part-0.parquet' source = s3.open_input_file(path) parquet_file = ParquetFile(source)   # row_group metadata parquet_file.metadata.row_group(0)  <pyarrow._parquet.RowGroupMetaData object at 0x7f8f5edcda40> num_columns: 22 num_rows: 11624 total_byte_size: 712185   # column_chunk metadata parquet_file.metadata.row_group(0).column(0)  <pyarrow._parquet.ColumnChunkMetaData object at 0x7f8f5edcda90> file_offset: 1636 file_path: physical_type: BYTE_ARRAY num_values: 11624 path_in_schema: vendor_name is_stats_set: True statistics: <pyarrow._parquet.Statistics object at 0x7f8f5eb74c20> has_min_max: True min: CMT max: VTS null_count: 0 distinct_count: 0 num_values: 11624 physical_type: BYTE_ARRAY logical_type: String converted_type (legacy): UTF8 compression: SNAPPY encodings: ('PLAIN_DICTIONARY', 'PLAIN', 'RLE') has_dictionary_page: True dictionary_page_offset: 4 data_page_offset: 41 total_compressed_size: 1632 total_uncompressed_size: 1625  But I cannot go further than that. Is it possible to get page related information (page header, repetition levels, definition levels and values) as outlined in parquet docs?  Note: I am interested in this to learn about how parquet files work under the hood. I've had a look at introspection tools (like parquet-tools) but it seems to be deprecated and alternatives only give row group level information.

Answer: You cannot access that information in pyarrow today. Pyarrow has initally been focused on converting parquet files to the Arrow representation. There is no equivalent to pages in Arrow. The info should be available in parquet-cpp (which, confusingly, is a project that also lives in the Arrow GitHub repo) if you're able to dig into C++. It may be possible to get that info in other parquet projects, I am not as familiar with them.
Question: I have a pandas dataframe df with two columns (type and IR) as this one:  type IR 0 a 0.1 1 b 0.3 2 b 0.2 3 c 0.8 4 c 0.5 ...  I want to plot three distributions (one for each type) with the values of the IR so, I write: sns.displot(df, kind="kde", hue='type', rug=True)  but I get this error: The following variable cannot be assigned with wide-form data 'hue' Any idea?  EDIT: My real dataframe looks like pd.DataFrame({"type": ["IR orig", "IR orig", "IR orig", "IR trans", "IR trans", "IR trans", "IR perm", "IR perm", "IR perm", "IR perm", "IR perm"], "IR": [1.41, 1.42, 1.32, 0.0, 0.44, 0.0, 1.41, 1.31, 1.41, 1.37, 1.34] })  but with sns.displot(df, x='IR', kind="kde", hue='type', rug=True) I got ValueError: cannot reindex on an axis with duplicate labels

Answer: Use: df = pd.DataFrame({'type': {0: 'a', 1: 'b', 2: 'b', 3: 'c', 4: 'c'}, 'IR': {0: 0.1, 1: 0.3, 2: 0.2, 3: 0.8, 4: 0.5}}) print (df) type IR 0 a 0.1 1 b 0.3 2 b 0.2 3 c 0.8 4 c 0.5 sns.displot(df.reset_index(drop=True), x='IR', kind="kde", hue='type', rug=True)
Question: I have a pandas data frame similar to this:     name sales profit profit_flag     Joe 200 100 True   Joe 300 150 False   Mark 200 100 True   Mark 300 150 True   Judy 300 150 False     The actual data frame has 100 columns. The idea is: I want to group by name, and aggregate all the columns. However, certain columns depend on a flag. In this case, sales will be aggregated no matter what, but profit should be included in the aggregation only if profit_flag is True. It should look like this if we use sum:     name sales profit     Joe 500 100   Judy 300 Nan   Mark 500 250     Is there anyway I can do this from one line using df.groupby('name').agg()? Right now I'm using: grouped = pd.DataFrame() grouped['sales'] = df.groupby('name').sales.sum() grouped['profit'] = df[df.profit_flag].groupby('name').profit.sum()  I'm getting the correct result, but since the actual data frame has many more columns, I wanted to know if I could somehow write something like this to avoid the clutter: grouped = df.groupby('name').agg({ 'sales': 'sum', 'profit' 'sum' #if profit_flag })  Is this even possible or should I just group 'flag dependent columns' in separate statemetns?

Answer: You can mask the values prior to aggregation: (df.assign(profit=lambda d: d['profit'].where(d['profit_flag'])) .groupby('name', as_index=False)[['sales', 'profit']].sum(min_count=1) )  Output:  name sales profit 0 Joe 500 100.0 1 Judy 300 NaN 2 Mark 500 250.0
Question: I have the numpy array arr = np.array([[0, 0, 2, 5, 0, 0, 1, 8, 0, 3, 0], [1, 2, 0, 0, 0, 0, 5, 7, 0, 0, 0], [8, 5, 3, 9, 0, 1, 0, 0, 0, 0, 1]])  I need the result array like this: [[0, 0, 0, 0, 7, 0, 0, 0, 9, 0, 3] [0, 0, 3, 0, 0, 0, 0, 0, 12, 0, 0] [0, 0, 0, 0, 25, 0, 1, 0, 0, 0, 0]]  What's happened? We go along the row, if element in row is 0, then we go to the next element , if not 0, then we sum up the elements until 0 is met, once 0 is met, then we replace it with the resulting sum (also replace the initial non-zero numbers with 0 I already know how to do that with loops but it doesn't work well on time for a large number of rows, so I need time-efficient solution in numpy methods

Answer: First, we want to find the locations where the array has a zero next to a non-zero. rr, cc = np.where((arr[:, 1:] == 0) & (arr[:, :-1] != 0))  Now, we can use np.add.reduceat to add elements. Unfortunately, reduceat needs a list of 1-d indices, so we're going to have to play with shapes a little. Calculating the equivalent indices of rr, cc in a flattened array is easy: reduce_indices = rr * arr.shape[1] + cc + 1 # array([ 4, 8, 10, 13, 19, 26, 28])  We want to reduce from the start of every row, so we'll create a row_starts to mix in with the indices calculated above: row_starts = np.arange(arr.shape[0]) * arr.shape[1] # array([ 0, 11, 22]) reduce_indices = np.hstack((row_starts, reduce_indices)) reduce_indices.sort() # array([ 0, 4, 8, 10, 11, 13, 19, 22, 26, 28])  Now, call np.add.reduceat on the flattened input array, reducing at reduce_indices totals = np.add.reduceat(arr.flatten(), reduce_indices) # array([ 7, 9, 3, 0, 3, 12, 0, 25, 1, 1])  Now we have the totals, we need to assign them to an array of zeros. Note that the 0th element of totals needs to go to the 1th index of reduce_indices, and the last element of totals is to be discarded: result_f = np.zeros((arr.size,)) result_f[reduce_indices[1:]] = totals[:-1] result = result_f.reshape(arr.shape)  Now, one last step remains. For cases where the last element in a row is nonzero, reduceat would calculate a nonzero value for the first element of the next row, as you mentioned in the comment below. An easy solution is to overwrite these to zero. result[:, 0] = 0  which gives the expected result: array([[ 0., 0., 0., 0., 7., 0., 0., 0., 9., 0., 3.], [ 0., 0., 3., 0., 0., 0., 0., 0., 12., 0., 0.], [ 0., 0., 0., 0., 25., 0., 1., 0., 0., 0., 0.]])
Question: I have this data frame: df:     id hint     29c45630-7d41-11e9-8ea2-9f2bfe5760ab None   61afc910-3918-11ea-b078-93fcef773138 Yes     and I want to find the first row that has this id = 29c45630-7d41-11e9-8ea2-9f2bfe5760ab when I run this code: df[df['id'] == '29c45630-7d41-11e9-8ea2-9f2bfe5760ab'] return empty!!! and the type of this id is "object" but I really have this id!! how can I find it?

Answer: You code didn't work because you have UUIDs, not strings, you first need to convert to string: df[df['id'].astype(str).eq('29c45630-7d41-11e9-8ea2-9f2bfe5760ab')]  Output:  id hint 0 29c45630-7d41-11e9-8ea2-9f2bfe5760ab None
Question: Given this Jupyter notebook cell: x = [1,2,3,4,5] y = {1,2,3,4,5} x y  When the cell executes, it generates this output: {1, 2, 3, 4, 5}  The last line in the cell generates output, the line above it has no effect. This works for any data type, as far as I can tell. Here's a snip of the same code as above: 

Answer: You can change this behaviour with: from IPython.core.interactiveshell import InteractiveShell InteractiveShell.ast_node_interactivity = "all"  The reason why only the last line is printed is that the default value of ast_node_interactivity is: last_expr. You can read more about that here: https://ipython.readthedocs.io/en/stable/config/options/terminal.html
Question: I would like to read several png images by utilizing the ThreadPoolExecutor and cv2.imread. Problem is that I don't know where to place cv2.IMREAD_UNCHANGED tag/argument to preserve alpha channel (transparency). The following code works but alpha channel is lost. Where should I place the cv2.IMREAD_UNCHANGED argument? import cv2 import concurrent.futures images=["pic1.png", "pic2.png", "pic3.png"] images_list=[] with concurrent.futures.ThreadPoolExecutor() as executor: images_list=list(executor.map(cv2.imread,images))  For example, the following return an error: SystemError: <built-in function imread> returned NULL without setting an error import cv2 import concurrent.futures images=["pic1.png", "pic2.png", "pic3.png"] images_list=[] with concurrent.futures.ThreadPoolExecutor() as executor: images_list=list(executor.map(cv2.imread(images,cv2.IMREAD_UNCHANGED))) 

Answer: Use a lambda that accepts one argument img and pass the argument to the imread function along with the cv2.IMREAD_UNCHANGED.  import cv2 import concurrent.futures images=["pic1.png", "pic2.png", "pic3.png"] images_list=[] with concurrent.futures.ThreadPoolExecutor() as executor: images_list=list(executor.map(lambda img: cv2.imread(img, cv2.IMREAD_UNCHANGED),images))
Question: I have trying to learn Python on datacamp. When I run statistical codes with agg - like min, max - I realized that it is the same result with using np with these codes. But I can not use mean and median methods without np. So,  is there any difference between np.min and min , np.max and max?  why don't median and mean work without numpy?   Code: unemp_fuel_stats = sales.groupby("type")["unemployment", "fuel_price_usd_per_l"].agg([np.min, np.max, np.mean, np.median])  Result: unemployment fuel_price_usd_per_l amin amax mean median amin amax mean median type A 3.879 8.992 7.973 8.067 0.664 1.107 0.745 0.735 B 7.170 9.765 9.279 9.199 0.760 1.108 0.806 0.803  Another result: unemployment fuel_price_usd_per_l min max min max type A 3.879 8.992 0.664 1.107 B 7.170 9.765 0.760 1.108 

Answer: min is the builtin python function and numpy.min is another min function that is in the numpy package. They act similarly but different. Given a single list, they will act the same: print(np.min([1, 2, 3, 4, 5])) print(min([1, 2, 3, 4, 5])) 1 1  However, given multiple arguments, numpy.min will throw an error as the first argument must be array-like. min will simply return the minimum of all elements. print(min(1, 2, 3, 4, 5)) try: print(np.min(1, 2, 3, 4, 5)) except Exception as e: print(e) 1 output must be an array  numpy.min can take the minimum along a specific axis of a multi-dimensional array (which uses multiple square brackets) or the minimum of all elements. min will simply given the minimum list (though I forget how they're ordered). twod_array = [[1, 2, 5], [3, 7, 5], [10, 5, 7]] print(min(twod_array)) print(np.min(twod_array)) print(np.min(twod_array, axis = 0)) # Minimum of each column print(np.min(twod_array, axis = 1)) # Minimum of each row [1, 2, 5] 1 [1 2 5] [1 3 5]  Hope that was enlightening. The tl;dr is that they are simply different functions with the same name. It's good to use import numpy as np to differentiate min from np.min. Using something like from numpy import min will cause ambiguity as to which function is actually being used. Python has no builtin median or mean but numpy has numpy.median and numpy.mean
Question: I am working with an ex termly large datfarem. Here is a sample: import pandas as pd import numpy as np df = pd.DataFrame({ 'ID': ['A', 'A', 'A', 'X', 'X', 'Y'], }) ID 0 A 1 A 2 A 3 X 4 X 5 Y  Now, given the frequency of each value in column '''ID''', I want to calculate a weight using the function below and add a column that has the weight associated with each value in '''ID'''. def get_weights_inverse_num_of_samples(label_counts, power=1.): no_of_classes = len(label_counts) weights_for_samples = 1.0/np.power(np.array(label_counts), power) weights_for_samples = weights_for_samples/ np.sum(weights_for_samples)*no_of_classes return weights_for_samples freq = df.value_counts() print(freq) ID A 3 X 2 Y 1 weights = get_weights_inverse_num_of_samples(freq) print(weights) [0.54545455 0.81818182 1.63636364]  So, I am looking for an efficient way to get a dataframe like this given the above weights:  ID sample_weight 0 A 0.54545455 1 A 0.54545455 2 A 0.54545455 3 X 0.81818182 4 X 0.81818182 5 Y 1.63636364 

Answer: If you rely on duck-typing a little bit more, you can rewrite your function to return the same input type as outputted. This will save you of needing to explicitly reaching back into the .index prior to calling .map import pandas as pd df = pd.DataFrame({'ID': ['A', 'A', 'A', 'X', 'X', 'Y'}) def get_weights_inverse_num_of_samples(label_counts, power=1): """Using object methods here instead of coercing to numpy ndarray""" no_of_classes = len(label_counts) weights_for_samples = 1 / (label_counts ** power) return weights_for_samples / weights_for_samples.sum() * no_of_classes # select the column before using `.value_counts()` # this saves us from ending up with a `MultiIndex` Series freq = df['ID'].value_counts() weights = get_weights_inverse_num_of_samples(freq) print(weights) # A 0.545455 # X 0.818182 # Y 1.636364 # note that now our weights are still a `pd.Series` # that we can align directly against our `"ID"` column df['sample_weight'] = df['ID'].map(weights) print(df) # ID sample_weight # 0 A 0.545455 # 1 A 0.545455 # 2 A 0.545455 # 3 X 0.818182 # 4 X 0.818182 # 5 Y 1.636364
Question: So, I have been trying to find optimum solution for the question, but I can not find a solution which is less than o(n3). The problem statemnt is :- find total number of triplet in an array such that sum of a[i],a[j],a[k] is divisible by a given number d and i<j<k. I have tried a multiple solutions but the solutions all reached o(n3). I need a solution that could be less than o(n3)

Answer: Let A be an array of numbers of length N: A = [1,2,3,4,5,6,7,8]  Let D be the divider D = 4  It is possible to reduce complexity O(N^2) with an extra dictionary that saves you iterating through the array for each pair (a[i],a[j]). The helper dictionary will be built before iterating through the pairs (i,j) with the count of A[k] % D = X. So for any pair A[i], A[j] you can tell how many matching A[k] exist by fetching from a dictionary rather than a loop. Below is a python implementation that demonstrates the solution T = 0 # Total possibilities H = {} # counts all possible (A[k])%D = Key from index k for k in range(2, len(A)): key = A[k]%D H[key] = H.get(key,0) + 1 for j in range(1, len(A)): if j >= 2: H[A[j]%D] -= 1 # when j increments it reduces options for A[k] for i in range(j): matching_val = (D - (A[i]+A[j]) % D ) % D to_add = H.get(matching_val, 0) T += to_add print(T)
Question: I want to use debian:bullseye as a base image and then install a specific Python version - i.e. 3.11.1. At the moment I am just learning docker and linux. From what I understand I can either:  Download and compile sources Install binaries (using apt-get) Use a Python base image  I have come across countless questions on here and articles online. Do I use deadsnakes? What version do I need? Are there any official python distributions (who is deadsnakes anyway)? But ultimately I want to know the best means of getting Python on there. I don't want to use a Python base image - I am curious in the steps involved. Compile sources - I am far from having that level of knowhow - and one for another day. Currently I am rolling with the following: FROM debian:bullseye RUN apt update && apt upgrade -y RUN apt install software-properties-common -y RUN add-apt-repository "ppa:deadsnakes/ppa" RUN apt install python3.11  This fails with: #8 1.546 E: Unable to locate package python3.11 #8 1.546 E: Couldn't find any package by glob 'python3.11'  Ultimately - it's not the error - its just finding a good way of getting a specific Python version on my container.

Answer: In case you want to install Python 3.11 in debian bullseye you have to compile it from source following the next steps (inside the Dockerfile): sudo apt update sudo apt install software-properties-common wget wget https://www.python.org/ftp/python/3.11.1/Python-3.11.1.tar.xz sudo tar -xf Python-3.11.1.tar.xz cd Python-3.11.1 sudo ./configure --enable-optimizations sudo make altinstall   Another option (easiest) would be to use the official Python Docker image, in your case: FROM 3.11-bullseye  You have all the versions available in docker hub. Other option that could be interesting in your case is 3.11-slim-bullseye, that is an image that does not contain the common packages contained in the default tag and only contains the minimal packages needed to run python.
Question: I have a pandas dataframe(sample). id name 1 Mr-Mrs-Jon Snow 2 Mr-Mrs-Jane Smith 3 Mr-Mrs-Darth Vader  I'm looking to strip the "Mr-Mrs-" from the dataframe. i.e the output should be: id name 1 Jon Snow 2 Jane Smith 3 Darth Vader  I tried using df['name'] = df['name'].str.lstrip("Mr-Mrs-") But while doing so, some of the alphabets of names in some rows are also getting stripped out. I don't want to run a loop and do .loc for every row, is there a better/optimized way to achieve this ?

Answer: Don't strip, replace using a start of string anchor (^): df['name'] = df['name'].str.replace(r"^Mr-Mrs-", "", regex=True)  Or removeprefix: df['name'] = df['name'].str.removeprefix("Mr-Mrs-")  Output: id name 1 Jon Snow 2 Jane Smith 3 Darth Vader
Question: I have got a list: first = [100, 110, 60]  How to make that: if the next number is greater than the previous one, then it is necessary to reduce this number like preceding number. For example, the answer should be: ans = [100, 100, 60]  The second example: arr = [60,50,60] ans = [60, 50, 50]  The third example: arr = [20, 100, 150] ans = [20, 20, 20]  I try to, but i think it's not good idea for i in range(len(arr)-1): if arr[i] < arr[i+1]: answer.append(a[i+1] - 10) if arr[i] < arr[i+1]: answer.append(a[i]) if arr[i] < arr [i+1]: answer.append(arr[-1]) 

Answer: This will modify the list in situ: def fix_list(_list): for i, v in enumerate(_list[1:], 1): _list[i] = min(v, _list[i-1]) return _list print(fix_list([100, 110, 60])) print(fix_list([60, 50, 60])) print(fix_list([20, 100, 150]))  Output: [100, 100, 60] [60, 50, 50] [20, 20, 20]