eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have created a docker image with the Docker file below. It installs the latest versions of Google Chrome and the chrome driver. As well as the other pip packages. Dockerfile FROM python:3.9 # Install Chrome WebDriver RUN CHROMEDRIVER_VERSION=`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE` && \ mkdir -p /opt/chromedriver-$CHROMEDRIVER_VERSION && \ curl -sS -o /tmp/chromedriver_linux64.zip http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip && \ unzip -qq /tmp/chromedriver_linux64.zip -d /opt/chromedriver-$CHROMEDRIVER_VERSION && \ rm /tmp/chromedriver_linux64.zip && \ chmod +x /opt/chromedriver-$CHROMEDRIVER_VERSION/chromedriver && \ ln -fs /opt/chromedriver-$CHROMEDRIVER_VERSION/chromedriver /usr/local/bin/chromedriver # Install Google Chrome RUN curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \ echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list && \ apt-get -yqq update && \ apt-get -yqq install google-chrome-stable && \ rm -rf /var/lib/apt/lists/* COPY requirements.txt . RUN pip install -r requirements.txt WORKDIR /seltesting COPY ./app ./app CMD ["python", "./app/main.py"]  The chromedriver.exe file is in the container as I have found it in the CLI. It is in this directory '/usr/local/bin/chromedriver'. python code driver = webdriver.Chrome(options=options, executable_path='/usr/local/bin/chromedriver')  I am using a venv as I am also using flask to create a micro service that uses the chrome driver. Would that be causing an issue? Any assist would be much appreciated as I have been stuck on this for a long time.

Answer: I have found the problem, you need to add the all the python files into the Dockerfile. Please find the Dockerfile to install the Chromedriver and Chrome onto the image and the default path for the chromedriver within the container. Dockerfile FROM python:3.9 ADD /app/main.py . ADD /app/connectdriver.py . # Install Chrome WebDriver RUN CHROMEDRIVER_VERSION=`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE` && \ mkdir -p /opt/chromedriver-$CHROMEDRIVER_VERSION && \ curl -sS -o /tmp/chromedriver_linux64.zip http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip && \ unzip -qq /tmp/chromedriver_linux64.zip -d /opt/chromedriver-$CHROMEDRIVER_VERSION && \ rm /tmp/chromedriver_linux64.zip && \ chmod +x /opt/chromedriver-$CHROMEDRIVER_VERSION/chromedriver && \ ln -fs /opt/chromedriver-$CHROMEDRIVER_VERSION/chromedriver /usr/local/bin/chromedriver # Install Google Chrome RUN curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \ echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list && \ apt-get -yqq update && \ apt-get -yqq install google-chrome-stable && \ rm -rf /var/lib/apt/lists/* COPY requirements.txt . RUN pip install -r requirements.txt WORKDIR /seleniumtesting COPY ./app ./app CMD ["python","./app/main.py"]  python driver script driver = webdriver.Chrome(options=options, executable_path='/usr/local/bin/chromedriver')
Question: Is it possible to iterate over attributes of a instance of dataclass in python? For example, I would like in the __post_init__ double the integer attributes: from dataclasses import dataclass, fields @dataclass class Foo: a: int b: int def __post_init__(self): self.double_attributes() def double_attributes(self): for field in fields(Foo): field = field*2 x = { 'a': 1, 'b': 2 } y = Foo(**x) >>> TypeError: unsupported operand type(s) for *: 'Field' and 'int'  How to access value of instances of a class and set it to something else like below but in a loop? @dataclass class Foo: a: int b: int def __post_init__(self): self.double_a() self.double_b() def double_a(self): self.a = self.a*2 def double_b(self): self.b = self.b*2 

Answer: You are very close, but dataclasses.fields actually returns a tuple of Field objects. At least in my case, it looks like the return type is not properly annotatted, but that's easy enough to fix. from dataclasses import dataclass, fields, Field from typing import Tuple @dataclass class Foo: a: int b: int def __post_init__(self): self.double_attributes() def double_attributes(self): # Note the annotation added here (tuple of one or more # `dataclasses.Field`s) cls_fields: Tuple[Field, ...] = fields(self.__class__) for field in cls_fields: # This check is to avoid fields annotated with other types # such as `str` if issubclass(field.type, int): new_val = getattr(self, field.name) * 2 setattr(self, field.name, new_val)  But if you're running this multiple times (for example creating many Foo objects) then it might be slightly efficient to cache the list of fields which are integers. For example the following is pseudo code which I might suggest: integer_fields: ClassVar[Frozenset[Field]] = frozenset(f for f in fields(cls) if issubclass(f.type, int))
Question: in Django PUT method, I want to update an instance: sv= SV.objects.get(pk=pk) serializer = SVSerializer(sv, data=request.data) if serializer.is_valid():  Here, in request.data, I just want to pass some of the variable of SV. But as some fields missing, the is_vaild will be false. What I want is, just update the fields in request.data, for the other ones, keep the value in sv. How could I do that?

Answer: Perform a partial update by setting partial=True: sv= SV.objects.get(pk=pk) serializer = SVSerializer(sv, data=request.data, partial=True) if serializer.is_valid(): serializer.save() else: # Do something else  This allows a PATCH request. Edit If you want a default field during partial update (as requested in a comment) override the update method: SVSerializer(serializers.ModelSerializer): # Instead of exposing the state_flag = serializers.SlugRelatedField(source='sv_state', queryset=SVState.objects.all(), slug_field='flag') def update(self, instance, validated_data): if self.partial and validated_data.get('state_flag') == None: validated_data['state_flag'] = 0 super().update(instance=instance, validated_data=validated_data)
Question: For example, we have a class: class A: def send(msg: bytes) -> None: # implementation... pass def recv(n: int) -> bytes: # implementation pass  and a function: def a(obj, n: int) -> None: received = obj.recv(n) obj.send(received)  It's fairly obvious, that not only instances of class A can be passed as the obj argument, but also instances of socket.socket, maybe other classes, which have recv and send implemented. How can one annotate/type hint obj argument, so that it says something like: obj type must possess methods send and recv send method must be of type Callable[[bytes], None] recv method must be of type Callable[[int], bytes] 

Answer: What you exactly need is duck-typing (structural subtyping) via typing.Protocol. Some examples are in this list.  Protocol classes are defined like this: class Proto(Protocol): def meth(self) -> int: ...  Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing), for example: class C: def meth(self) -> int: return 0 def func(x: Proto) -> int: return x.meth() func(C()) # Passes static type check   Where a builtin example is  class typing.SupportsIndex An ABC with one abstract method __index__.  So for your case, it could be something like: from typing import Protocol class SupportsSendReceive(Protocol): def send(self, msg: bytes) -> None: ... def recv(self, n: int) -> bytes: ... def a(obj: SupportsSendReceive, n: int) -> None: received = obj.recv(n) obj.send(received)   Note that the ellipsis ... doesn't mean you have to substitute code into it. It is really how it should be. Or you could also put pass in there if the 3 dots are bothering :)
Question: We have a pyspark dataframe with several columns containing arrays with multiple values. Our goal is to have each of this values of these columns in several rows, keeping the initial different columns. So, starting with something like this: data = [ ("A", ["a", "c"], ["1", "5"]), ("B", ["a", "b"], None), ("C", [], ["1"]), ]  Whats: +---+------+------+ |id |list_a|list_b| +---+------+------+ |A |[a, c]|[1, 5]| |B |[a, b]|null | |C |[] |[1] | +---+------+------+  We would like to end up having: +---+----+----+ |id |col |col | +---+----+----+ |A |a |null| |A |c |null| |A |null|1 | |A |null|5 | |B |a |null| |B |b |null| |C |null|1 | +---+----+----+  We are thinking about several approaches:  prefixing each value with a column indicator, merge all the arrays into a single one, explode it and reorganize the different values into different columns split the dataframe into several, each one with one of these array columns, explode the array column and then, concatenating the dataframes  But all of them smell like dirty, complex, error prone and inefficient workarounds. Does anyone have an idea about how to solve this in an elegant manner?

Answer: In case both columns list_a and list_b could be empty, I would add a 4th case in the dataset data = [ ("A", ["a", "c"], ["1", "5"]), ("B", ["a", "b"], None), ("C", [], ["1"]), ("D", None, None), ] df = spark.createDataFrame(data,["id","list_a","list_b"])  I would then split the original df in 3 (both nulls, list_a exploded and list_b exploded) and the execute a unionByName dfnulls = df.filter(col("list_a").isNull() & col("list_b").isNull())\ .withColumn("list_a", lit(None))\ .withColumn("list_b", lit(None)) df1 = df\ .withColumn("list_a", explode_outer(col("list_a")))\ .withColumn("list_b", lit(None))\ .filter(~col("list_a").isNull()) df2 = df\ .withColumn("list_b", explode_outer(col("list_b")))\ .withColumn("list_a", lit(None))\ .filter(~col("list_b").isNull()) merged_df = df1.unionByName(df2).unionByName(dfnulls) merged_df.show() +---+------+------+ | id|list_a|list_b| +---+------+------+ | A| a| null| | A| c| null| | B| a| null| | B| b| null| | A| null| 1| | A| null| 5| | C| null| 1| | D| null| null| +---+------+------+
Question: The documentation shows how to do it with mongosh, but how do you create Time Series Collection using pymongo from within a python script? import pymongo import time from datetime import datetime client = pymongo.MongoClient() db = client['time-series-db'] col = db['time-series-col'] # ... do something here to make it 'time-series collection' ... js = { "1": "A", "2": "B", "3": "C", "4": "D", "5": "E", } # create BSON type datetime object needed for 'time-series collection' ts = time.time() js['timestamp'] = datetime.utcfromtimestamp(ts) col.insert_one(js) 

Answer: You can try this: conn = pymongo.MongoClient('mongodb://localhost') db = conn.testDB db.create_collection('testColl', timeseries={ 'timeField': 'timestamp' }) # - OR - db.command('create', 'testColl', timeseries={ 'timeField': 'timestamp', 'metaField': 'data', 'granularity': 'hours' })  General Reference: Time Series Collections
Question: I have a pandas DataFrame which has a column of dtype datetime: import pandas as pd # Mock-up data df = pd.DataFrame({'year': [2015, 2016], 'month': [2, 3], 'day': [4, 5]}) df = pd.to_datetime(df) print(df) # 0 2015-02-04 # 1 2016-03-05 # dtype: datetime64[ns]  I would like to use the .to_markdown() method to display this DataFrame. However, the .to_markdown() method displays the datetimes in scientific notation: print(df.to_markdown()) # | | 0 | # |---:|------------:| # | 0 | 1.42301e+18 | # | 1 | 1.45714e+18 |  Is there a way to have the .to_markdown() method display these dates in a more human-readable manner? The .to_latex(), .to_csv(), and .to_string() methods already behave this way: # Other .to_ methods behave as desired, eg. print(df.to_latex()) # \begin{tabular}{ll} # \toprule # {} & 0 \\ # \midrule # 0 & 2015-02-04 \\ # 1 & 2016-03-05 \\ # \bottomrule # \end{tabular}   pandas version: 1.3.2 tabulate version: 0.8.9

Answer: Under the hood the .to_markdown() method uses the tabulate package. The floatfmt named argument can be used to control the formatting of floats, but I cannot see how this could be useful here. The best solution I can currently find is simply to format the datetime column as a column of strings before calling the .to_markdown() method: print(df.astype(str).to_markdown()) # | | 0 | # |---:|:-----------| # | 0 | 2015-02-04 | # | 1 | 2016-03-05 |
Question: I want to define a dict variable once, generated from a text file, and use it to answer to API requests. This variable should be always available till the end of server run. In an example below: from fastapi import FastAPI import uvicorn app = FastAPI() def init_data(path): print("init call") data = {} data[1] = "123" data[2] = "abc" return data data = init_data('path') @app.get('/') def example_method(): # data is defined return {'Data': data[1]} if __name__ == '__main__': uvicorn.run(f'example_trouble:app', host='localhost', port=8000)  I will get: init call init call INFO: Started server process [9356] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)  and request to localhost:8000 wouldn't raise any errors How should I define a variable once, that would be accessed as a global variable to any request? Is there a common way to define it once and use it? requirements if necessary: fastapi==0.68.1 pydantic==1.8.2 starlette==0.14.2 typing-extensions==3.10.0.2 

Answer: One approach would be to use the FastAPI startup event to define the variable data once on app startup. An example similar to what you provided in your question: from fastapi import FastAPI import uvicorn app = FastAPI() data = {} @app.on_event('startup') def init_data(): print("init call") path='/an/example/path' data[1] = "123" data[2] = "abc" return data @app.get('/') def example_method(): # data is defined return {'Data': data[1]} if __name__ == '__main__': uvicorn.run(f'example_trouble:app', host='localhost', port=8000)  When running the app, you'll see that function is only executed once: INFO: Started server process [37992] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) init call
Question: I have a list of data frames but in a few cases, the list can also contain a string. df = pd.DataFrame({"df_column":["df_value"]}) a = ['skip',df] if "skip" in a: print("yes")  The above gives output as yes because the list contains a string. But in case if the list doesn't contain a string for eg df = pd.DataFrame({"df_column":["df_value"]}) a = [df,df] if "skip" in a: print("yes")  Now the above code gives an error ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). How can I handle this?

Answer: You need to avoid checking pandas all together see e.g. [df , 'skip'] would fail - it's just a matter of order. For starters you can only filter strings in a: if "skip" in filter(lambda x: isinstance(x, str), a): print("yes")
Question: I have an array a, and want to create a new matrix A for which A[i,j] = a[i+j]. Example: import numpy as np a = np.random.rand(3) A = np.zeros((2,2)); for i in range(2): for j in range(2): A[i,j] = a[i+j]  Is there a way of doing this without a for loop? (with numpy)

Answer: Using stride_tricks.as_strided This would be a perfect use case for stride_tricks: from np.lib.stride_tricks import as_strided  Set the strides as (8, 8) (i.e. (1, 1) in terms of slots). This way we essentially map the resulting array A as i, j -> k = i + j. A more detailed description would be: we map every i, j pair to a natural number k defined by the strides as k = i*s_i + j*s_j, where s_i and s_j are the strides set to 1s, i.e. k = i + j. Thus ending up with the desired result: A[i, j] = a[k] = a[i + j]. >>> a array([0.53954179, 0.51789927, 0.33982179]) >>> A = as_strided(a, shape=(2,2), strides=(8,8)) array([[0.53954179, 0.51789927], [0.51789927, 0.33982179]])  Additional considerations A more general solution would be to get the shapes as well as the strides from a's metadata.  The shape of A is given by (len(a)//2+1,)*2.  As noted by @Daniel F, the memory slot size does not always equal to 8, this indeed depends on the dtype of your array. It would be better to define strides from a's strides instead: a.strides*2.    This comes down to: >>> A = as_strided(a, shape=(len(a)//2+1,)*2, strides=a.strides*2)   Using grid indexing Alternatively, you can construct a grid of coordinates (you can do so using itertools.product) then copy the appropriate values from a to A: i, j = np.array(list(product(range(2), range(2)))).T  Then initialize A and copy: >>> A = np.zeros((2,2)) >>> A[i, j] = a[i + j]  This will however double the memory usage, compared to the as_strided method.
Question: I'm facing an issue with grouping and transforming on non-NA values in my dataframe. So my dataframe is something like this:     Name Value     A 1   A 2   A NaN   B 3   B 7   B 9   B NaN     Final output I want:     Name Value Weight 1 Weight 2     A 1 0.33 0.5   A 2 0.33 0.5   A NaN 0.33 NaN   B 3 0.25 0.33   B 7 0.25 0.33   B 9 0.25 0.33   B NaN 0.25 NaN     I know this may sound trivial but I'm not able to get the Weight 2 working perfectly across different grouping categories of column Name. Here's how I get column Weight 1: df['Weight 1'] = df.groupby(['Name']).transform(lambda x: 1/len(x))  So far I tried following on Weight 2, but raises DivisionByZero warning. Output is incorrect. df['Weight 2'] = df.groupby(['Name']).transform(lambda x: 1/np.sum(~np.isnan(x)))  Any help is appreciated.

Answer: You can use GroupBy.count to count Non-NaN values in each group. Then use pd.Series.map with pd.Series.mask mapping = (1 / df.groupby('Name')['Value'].count()).squeeze() df['Weight 2'] = df['Name'].map(mapping).mask(df['Value'].isna()) Name Value Weight 2 0 A 1.0 0.500000 1 A 2.0 0.500000 2 A NaN NaN 3 B 3.0 0.333333 4 B 7.0 0.333333 5 B 9.0 0.333333 6 B NaN NaN
Question: I have a below NormalClass that I want to structure as a dataclass. However I was not sure how I can pass the date_str param without __init__ in the dataclass. Any thoughts? class FieldDateTime(): def __init__(self, data, d_format='%m/%d/%y %I:%M %p'): try: self.data = datetime.strptime(data, d_format) except ValueError as e: raise ValueError('Dateformat incorrect') def __call__(self): return self.data class NormalClass: def __init__(self, id, date_str): self.id: int = id self.dt: FieldDateTime = FieldDateTime(date_str) @dataclass class DataClassObj: id: int dt: FieldDateTime(date_str)  How do I pass the date_str as an argument in the data class representation (DataClassObj) without the init?

Answer: Your question has to be more detailed, but I think this is what you're looking for: from __future__ import annotations from datetime import datetime from dataclasses import dataclass, InitVar, field class FieldDateTime: def __init__(self, data, d_format='%m/%d/%y %I:%M %p'): try: self.data = datetime.strptime(data, d_format) except ValueError as e: raise ValueError('Dateformat incorrect') def __repr__(self): return f"{self.__class__.__name__}({self.data})" def __call__(self): return self.data @dataclass class NormalDataClass: id: int date_str: InitVar[str] dt: FieldDateTime = field(init=False) def __post_init__(self, date_str): self.dt = FieldDateTime(date_str) print(NormalDataClass(10, '09/04/21 08:11 PM'))  output (given the dataclass-like __repr__ implementation on FieldDateTime to make it look a bit better): NormalDataClass(id=10, dt=FieldDateTime(2021-09-04 20:11:00))  from doc:  Init-only fields are added as parameters to the generated __init__ method, and are passed to the optional __post_init__ method.  So we can use InitVar for our date_str and pass it to __post_init__ to create our FieldDateTime object.
Question: I see a lot of questions like this one for R, but I couldn't find one specifically for Python, preferably using numpy. Let's say I have an array of observations stored in x. I can get the value that accumulates q * 100 per cent of the population. # Import numpy import numpy as np # Get 75th percentile np.quantile(a=x, q=0.75)  However, I was wondering if there's a function that does the inverse. That is, a numpy function that takes a value as an input and returns q. To further expand on this, scipy distribution objects have a ppf method that allows me to do this. I'm looking for something similar in numpy. Does it exist?

Answer: Not a ready-made function but a compact and reasonably fast snippet: (a<value).mean()  You can (at least on my machine) squeeze out a few percent better performance by using np.count_nonzero np.count_nonzero(a<value) / a.size  but tbh I wouldn't even bother.
Question: I was wondering if we could assert all elements in a list is not None, therefore while a = None will raise an error. The sample list is [a, b, c] I have tried assert [a, b, c] is not None, it will return True if any one of the elements is not None but not verifying all. Could you help figure it out? Thanks!!

Answer: Unless you have a weird element that claims it equals None: assert None not in [a, b, c]
Question: distplot was deprecated in favour of displot. The previous function had the option to draw a normal curve. import seaborn as sns import matplotlib.pyplot as plt from scipy import stats ax = sns.distplot(df.extracted, bins=40, kde=False, fit=stats.norm)  the fit=stats.norm doesn't work with displot anymore. In the answer to this question, I see the approach to plot the normal later, however it is done on some random data averaged around 0.

Answer:  seaborn.displot is a figure-level plot where the kind parameter specifies the approach. When kind='hist' the parameters for seaborn.histplot are available.  For axes-level plots see How to add a standard normal pdf over a seaborn histogram   seaborn.axisgrid.FacetGrid.map expects dataframe column names, as such, to map the pdf onto seaborn.displot, the data needs to be in a dataframe. An issue is that x_pdf is calculated for each axes:  x0, x1 = p1.axes[0][0].get_xlim() If the axes are different for multiple Facets (sharex=False), then there's not a way to get xlim for each axes within .map.   References:  seaborn histplot and displot output doesn't match Building structured multi-plot grids   Tested in python 3.8.11, pandas 1.3.2, matplotlib 3.4.2, seaborn 0.11.2  Single Facet  .map can be used  import pandas as pd import seaborn as sns import numpy as np import scipy # data np.random.seed(365) x1 = np.random.normal(10, 3.4, size=1000) # mean of 10 df = pd.DataFrame({'x1': x1}) # display(df.head(3)) x1 0 10.570932 1 11.779918 2 12.779077 # function for mapping the pdf def map_pdf(x, **kwargs): mu, std = scipy.stats.norm.fit(x) x0, x1 = p1.axes[0][0].get_xlim() # axes for p1 is required to determine x_pdf x_pdf = np.linspace(x0, x1, 100) y_pdf = scipy.stats.norm.pdf(x_pdf, mu, std) plt.plot(x_pdf, y_pdf, c='r') p1 = sns.displot(data=df, x='x1', kind='hist', bins=40, stat='density') p1.map(map_pdf, 'x1')   Single or Multiple Facets  It's easier to iterate through each axes and add the pdf  # data np.random.seed(365) x1 = np.random.normal(10, 3.4, size=1000) # mean of 10 x2 = np.random.standard_normal(1000) # mean of 0 df = pd.DataFrame({'x1': x1, 'x2': x2}).melt() # create long dataframe # display(df.head(3)) variable value 0 x1 10.570932 1 x1 11.779918 2 x1 12.779077 p1 = sns.displot(data=df, x='value', col='variable', kind='hist', bins=40, stat='density', common_bins=False, common_norm=False, facet_kws={'sharey': True, 'sharex': False}) # extract and flatten the axes from the figure axes = p1.axes.ravel() # iterate through each axes for ax in axes: # extract the variable name var = ax.get_title().split(' = ')[1] # select the data for the variable data = df[df.variable.eq(var)] mu, std = scipy.stats.norm.fit(data['value']) x0, x1 = ax.get_xlim() x_pdf = np.linspace(x0, x1, 100) y_pdf = scipy.stats.norm.pdf(x_pdf, mu, std) ax.plot(x_pdf, y_pdf, c='r')