eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My input is a array of 64 integers. model = Sequential() model.add( Input(shape=(68,), name="input")) model.add(Conv1D(64, 2, activation="relu", padding="same", name="convLayer"))  I have 10,000 of these arrays in my training set. And I supposed to be specifying this in order for conv1D to work? I am getting the dreaded ValueError: Input 0 of layer convLayer is incompatible with the layer: : expected min_ndim=3, found ndim=2. Full shape received: [None, 68]  error and I really don't understand what I need to do.

Answer: Don't let the name confuse you. The layer tf.keras.layers.Conv1D needs the following shape: (time_steps, features). If your dataset is made of 10,000 samples with each sample having 64 values, then your data has the shape (10000, 64), which is not directly applicable to the tf.keras.layers.Conv1D layer. You are missing the time_steps dimension. What you can do is use the tf.keras.layers.RepeatVector, which repeats your array input n times, in the example 5. This way your Conv1D layer gets an input of the shape (5, 64). Check out the documentation for more information:  time_steps = 5 model = tf.keras.Sequential() model.add(tf.keras.layers.Input(shape=(64,), name="input")) model.add(tf.keras.layers.RepeatVector(time_steps)) model.add(tf.keras.layers.Conv1D(64, 2, activation="relu", padding="same", name="convLayer"))  As a side note, you should ask yourself if using a tf.keras.layers.Conv1D layer is the right option for your use case. This layer is usually used for NLP and other time series tasks. For example, in sentence classification, each word in a sentence is usually mapped to a high-dimensional word vector representation, as seen in the image. This results in data with the shape (time_steps, features).   If you want to use character one hot encoded embeddings it would look something like this:   This is a simple example of one single sample with the shape (10, 10) --> 10 characters along the time series dimension and 10 features. It should help you understand the tutorial I mentioned a bit better.
Question: Can anyone help me fix this error. I just started using dataclass I wanted to put a default value so I can easily call from other function I have this class @dataclass(frozen=True) class MyClass: my_list: list = ["list1", "list2", "list3"] my_list2: list = ["list1", "list2", "list3"]  But when i print print(MyClass.my_list) I'm getting this error  raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'list'> for field my_list is not allowed: use default_factory 

Answer: What it means by mutable default is that the lists provided as defaults will be the same individual objects in each instance of the dataclass. This would be confusing because mutating the list in an instance by e.g. appending to it would also append to the list in every other instance. Instead, it wants you to provide a default_factory function that will make a new list for each instance: from dataclasses import dataclass, field @dataclass class MyClass: my_list: list = field(default_factory=lambda: ["list1", "list2", "list3"]) my_list2: list = field(default_factory=lambda: ["list1", "list2", "list3"])
Question: I have an animated plotly scatter graph which plots x,y coordinates normally within the 0-0.5 range with date/time being the frame key. Sometime however I will have to handle anomalous data points which will be well out with this range. I would like the graph to be able to dynamically scale so that the points are not lost off screen. Is this possible? def draw(x1,y1,timestamp): d = { "x1": x1_trim, "y1": y1_trim, "time": time_trim } df = pd.DataFrame(d) fig = px.scatter(df, x="x1", y="y1", animation_frame="time") fig.update_yaxes(autorange=True) fig.update_xaxes(autorange=True) fig.show()  I've tried using update_x/yaxes with autorange but it doesn't seem to work.

Answer: As pointed out in the comments, there is a way to change a plotly animation axis scale per frame. The question remains how dynamic it's possible to make it. But if we can say that you've made a few calculations that will let you know which frames you'd like to adjust the ranges for, then a combination of a dict like yranges = {2002:[0, 200]} and a for-loop on the frames should do the trick. Let's take a subset of the dataset px.data.gapminder as an example. And let's say that you'd like to adjust the range of the yaxis for the frame that displays data for the year 2002 in the following figure:  Then you can include the following snippet, and get the figure below for year = 2002: yranges = {2002:[0, 200]} for f in fig.frames: if int(f.name) in yranges.keys(): f.layout.update(yaxis_range = yranges[int(f.name)])   Complete code: import plotly.express as px df = px.data.gapminder() df = df[(df['continent'] == 'Asia') & (df['year'].isin([1997, 2002, 2007]))] scales = [2002] fig = px.scatter(df, x="gdpPercap", y="lifeExp", animation_frame="year", animation_group="country", size="pop", color="continent", hover_name="country", log_x=True, size_max=55, range_x=[100,100000], range_y=[25,90]) yranges = {2002:[0, 200]} for f in fig.frames: if int(f.name) in yranges.keys(): f.layout.update(yaxis_range = yranges[int(f.name)]) fig.show()
Question: I am trying to create a new column 'Var' in the following Pandas DataFrame based on values from the other columns. I am encountering issues when dealing with the NaN, NaT. Data: ( Used apply(pd.to_datetime) on the Date columns at a previous step)     Date C A Age     2017-12-13 1233.0 N 9   NaT NaN N 5   2007-09-24 49.0 N 14     Code: def program(Flat): if Flat['A'] == 'N' : return 0 elif Flat['Date'].isna() : return Flat['Age'] + 1 elif Flat['C'] < 365 : return 1 elif Flat['C'] >= 365 : return math.floor((Flat['C'])/365.25) + 1 Flat['Var'] = Flat.apply(program, axis=1)   Error: AttributeError: 'NaTType' object has no attribute 'isna'  Tried running through Anaconda & Python. All same error. Pandas version is 1.3.3. What is the correct way to detect the NaT type?

Answer: "NaT" (for date/time types) and "NaN" are not the same. However, you can use the "isnull" function for both types:  elif pd.isnull(Flat['Data']):
Question: What is the fundamental difference between tarring a folder using tar on Unix and tarfile in Python that results in a different file size? In the example below, there is an 8.2 MB difference. I'm currently using a Mac. The folder in this example contains a bunch of random text files for testing purposes. tar -cvf archive_unix.tar files/ python -m tarfile -c archive_pycli.tar files/ # using Python 3.9.6 -rw-r--r-- 1 userid staff 24606720 Oct 15 09:40 archive_pycli.tar -rw-r--r-- 1 userid staff 16397824 Oct 15 09:39 archive_unix.tar 

Answer: Interesting question. The documentation of tarfile (https://docs.python.org/3/library/tarfile.html) mentions that the default format for tar archive created by tarfile is, since python 3.8, PAX_FORMAT whereas archives created by the tar command have the GNU format which I believe explains the difference. Now to produce the same archive as the tar command and one with the default format (as your command did): import tarfile with tarfile.TarFile(name='archive-py-gnu.tar', mode='w', format=tarfile.GNU_FORMAT) as tf: tf.add('tmp') with tarfile.TarFile(name='archive-py-default.tar', mode='w') as tf: tf.add('tmp')  For comparison: $ tar cf archive-tar.tar tmp/ $ ls -l 3430400 16:28 archive-py-default.tar 3317760 16:28 archive-py-gnu.tar 3317760 16:27 archive-tar.tar  Results of the file command: $ file archive_unix.tar archive_unix.tar: POSIX tar archive (GNU) $ file archive-py-gnu.tar archive-py-gnu.tar: POSIX tar archive (GNU) $ file archive-py-default.tar archive-py-default.tar: POSIX tar archive  Now I cannot tell you the difference between the different formats, sorry. But I hope this helps.
Question: I need to copy all files from one prefix in S3 to another prefix within the same bucket. My solution is something like: file_list = [List of files in first prefix] for file in file_list: copy_source = {'Bucket': my_bucket, 'Key': file} s3_client.copy(copy_source, my_bucket, new_prefix)  However I am only moving 200 tiny files (1 kb each) and this procedure takes up to 30 seconds. It must be possible to do it fasteer?

Answer: I would do it in parallel. For example: from multiprocessing import Pool file_list = [List of files in first prefix] print(objects_to_download) def s3_coppier(s3_file): copy_source = {'Bucket': my_bucket, 'Key': s3_file} s3_client.copy(copy_source, my_bucket, new_prefix) # copy 5 objects at the same time with Pool(5) as p: p.map(s3_coppier, file_list)
Question: PEP 622, Literal Patterns says the following:  Note that because equality (__eq__) is used, and the equivalency between Booleans and the integers 0 and 1, there is no practical difference between the following two: case True: ... case 1: ...   and True.__eq__(1) and (1).__eq__(True) both returns True, but when I run these two code snippets with CPython, it seems like case True and case 1 are not same. $ python3.10 >>> match 1: ... case True: ... print('a') # not executed ... >>> match True: ... case 1: ... print('a') # executed ... a  How are 1 and True actually compared?

Answer: Looking at the pattern matching specification, this falls under a "literal pattern":  A literal pattern succeeds if the subject value compares equal to the value expressed by the literal, using the following comparisons rules:  Numbers and strings are compared using the == operator. The singleton literals None, True and False are compared using the is operator.   So when the pattern is:  case True:  It uses is, and 1 is True is false. On the other hand, case 1:  Uses ==, and 1 == True is true.
Question: I am trying to overload several operators at once using the __getattr__ function. In my code, if I call foo.__add__(other) it works as expected, but when I try foo + bar, it does not. Here is a minimal example: class Foo(): def add(self, other): return 1 + other def sub(self, other): return 1 - other def __getattr__(self, name): stripped = name.strip('_') if stripped in {'sub', 'add'}: return getattr(self, stripped) else: return if __name__=='__main__': bar = Foo() print(bar.__add__(1)) # works print(bar + 1) # doesn't work  I realize that it would be easier in this example to just define __add__ and __sub__, but that is not an option in my case. Also, as a small side question, if I replace the line: if stripped in {'sub', 'add'}:  with if hasattr(self, name):  the code works, but then my iPython kernel crashes. Why does this happen and how could I prevent it?

Answer: This is happening because python operators use an optimization to look up the function implementing the operator. The following lines are roughly equivalent: foo + 1 type(foo).__add__(foo, 1)  Operators are found specifically on the class object only, never on the instance. bar.__add__(1) calls __getattr__ to find the missing attribute on bar. This works because it bypasses normal operator lookup procedures. bar + 1 calls Foo.__add__(bar, 1) followed by int.__radd(1, bar). The first attribute lookup fails, and the second option raises TypeError.
Question: By entering the file with its extension, my code succeeds to detect the type of the file from the "magic number". magic_numbers = {'png': bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]), 'jpg': bytes([0xFF, 0xD8, 0xFF, 0xE0]), #*********************# 'doc': bytes([0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1]), 'xls': bytes([0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1]), 'ppt': bytes([0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1]), #*********************# 'docx': bytes([0x50, 0x4B, 0x03, 0x04, 0x14, 0x00, 0x06, 0x00]), 'xlsx': bytes([0x50, 0x4B, 0x03, 0x04, 0x14, 0x00, 0x06, 0x00]), 'pptx': bytes([0x50, 0x4B, 0x03, 0x04, 0x14, 0x00, 0x06, 0x00]), #*********************# 'pdf': bytes([0x25, 0x50, 0x44, 0x46]), #*********************# 'dll': bytes([0x4D, 0x5A, 0x90, 0x00]), 'exe': bytes([0x4D, 0x5A]), } max_read_size = max(len(m) for m in magic_numbers.values()) with open('file.pdf', 'rb') as fd: file_head = fd.read(max_read_size) if file_head.startswith(magic_numbers['pdf']): print("It's a PDF File") else: print("It's not a PDF file")  I want to know how I can modify it without specifying this part of code, i.e. once I generate or I enter the file it shows me directly the type of the file. if file_head.startswith(magic_numbers['pdf']): print("It's a PDF File") else: print("It's not a PDF file")  I hope you understand me.

Answer: You most like just want to iterate over the loop and test them all. You may be able to optimize or provide some error checking by using the extension as well. If you strip off the extension and check that first, you'll be successful most of the time, and if not you may not want to accept "baby.png" as an xlsx file. That would be suspicious and worthy of an error. But, if you ignore extension, just loop over the entries: for ext in magic_numbers: if file_head.startswith(magic_numbers[ext]): print("It's a {} File".format(ext))  You probably want to put this in a function that returns the type, so you could just return the type instead of printing it out. EDIT Since some share magic numbers, we need to assume the extension is correct until we know that it isn't. I would extract the extension from the filename. This could be done with Pathlib or just string split: ext = filename.rsplit('.', 1)[-1]  then test it specifically if ext in magic_numbers: if file_head.startswith(magic_numbers[ext]): return ext  put the ext test first, so putting it all together: ext = filename.rsplit('.', 1)[-1] if ext in magic_numbers: if file_head.startswith(magic_numbers[ext]): return ext for ext in magic_numbers: if file_head.startswith(magic_numbers[ext]): return ext return nil
Question: This might be heavily related to similar questions as Python 3.3: Split string and create all combinations , but I can't infer a pythonic solution out of this. Question is: Let there be a str such as 'hi|guys|whats|app', and I need all permutations of splitting that str by a separator. Example: #splitting only once ['hi','guys|whats|app'] ['hi|guys','whats|app'] ['hi|guys|whats','app'] #splitting only twice ['hi','guys','whats|app'] ['hi','guys|whats','app'] #splitting only three times ... etc  I could write a backtracking algorithm, but does python (itertools, e.g.) offer a library that simplifies this algorithm? Thanks in advance!!

Answer: An approach, once you have split the string is to use itertools.combinations to define the split points in the list, the other positions should be fused again. def lst_merge(lst, positions, sep='|'): '''merges a list on points other than positions''' '''A, B, C, D and 0, 1 -> A, B, C|D''' a = -1 out = [] for b in list(positions)+[len(lst)-1]: out.append('|'.join(lst[a+1:b+1])) a = b return out def split_comb(s, split=1, sep='|'): from itertools import combinations l = s.split(sep) return [lst_merge(l, pos, sep=sep) for pos in combinations(range(len(l)-1), split)]  examples >>> split_comb('hi|guys|whats|app', 0) [['hi|guys|whats|app']] >>> split_comb('hi|guys|whats|app', 1) [['hi', 'guys|whats|app'], ['hi|guys', 'whats|app'], ['hi|guys|whats', 'app']] >>> split_comb('hi|guys|whats|app', 2) [['hi', 'guys', 'whats|app'], ['hi', 'guys|whats', 'app'], ['hi|guys', 'whats', 'app']] >>> split_comb('hi|guys|whats|app', 3) [['hi', 'guys', 'whats', 'app']] >>> split_comb('hi|guys|whats|app', 4) [] ## impossible  rationale ABCD -> A B C D 0 1 2 combinations of split points: 0/1 or 0/2 or 1/2 0/1 -> merge on 2 -> A B CD 0/2 -> merge on 1 -> A BC D 1/2 -> merge on 0 -> AB C D  generic function Here is a generic version, working like above but also taking -1 as parameter for split, in which case it will output all combinations def lst_merge(lst, positions, sep='|'): a = -1 out = [] for b in list(positions)+[len(lst)-1]: out.append('|'.join(lst[a+1:b+1])) a = b return out def split_comb(s, split=1, sep='|'): from itertools import combinations, chain l = s.split(sep) if split == -1: pos = chain.from_iterable(combinations(range(len(l)-1), r) for r in range(len(l)+1)) else: pos = combinations(range(len(l)-1), split) return [lst_merge(l, pos, sep=sep) for pos in pos]  example: >>> split_comb('hi|guys|whats|app', -1) [['hi|guys|whats|app'], ['hi', 'guys|whats|app'], ['hi|guys', 'whats|app'], ['hi|guys|whats', 'app'], ['hi', 'guys', 'whats|app'], ['hi', 'guys|whats', 'app'], ['hi|guys', 'whats', 'app'], ['hi', 'guys', 'whats', 'app']]
Question: We have a python package which is also a git repo. It depends on other python packages, themselves git repos. We made the latter git submodules of the former. None of these are public, so no PyPI. None of the other questions related to installing with submodule dependencies match our pattern. My question is not about finding (sub)packages with setuptools, nor is it about relative imports. This is our structure: package-repo/ setup.py setup.cfg README.md .gitignore .gitmodules .git/ submodule-repo/ .git/ .gitignore setup.py setup.cfg README.md submodule/ __init__.py moduleX.py moduleY.py package/ __init__.py moduleA.py moduleB.py subpackage1/  As is the case with requirements.txt, I naively though that something as follow would work out: from setuptools import setup setup(name='package', version='0.4.1', description='A package depending on other self made packages', url='git.ownnetwork.com', author='wli', author_email='wli@', license='Proprietary', packages=['package','package.subpackage1'], include_package_data=True, python_requires='>=3.7', classifiers=[ 'Natural Language :: English', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', ], install_requires=["SQLAlchemy", "pandas", "./submodule-repo"])  It doesn't work. An alternative is to add the submodule in packages and indicate its directory in package_dir. Well it didn't work that well, and what's the point of making a setup.py in the "submodule" if it can't be installed when installing the dependent module? I just want it to be installed without having to put it on PyPI or create a virtual PyPI server, which is seriously overkill, or having to indicate what to do (i.e. pip install ./submodule-repo/) in the README.md, which is inelegant. What's the way? Did I miss it in distutils or setuptools documentation?

Answer: You will need to specify where to install the submodule from. install_requires=[ 'SQLAlchemy', 'pandas', # Your private repository module '<dependency_name> @ git+ssh://git@github.com/<user_name>/<repo_name>@<branch>' ]
Question: I have the following code that uses Pydantic BaseModel data class from enum import Enum import requests from pydantic import BaseModel from requests import Response class PetType(Enum): DOG: str = 'dog' CAT: str = 'cat' class Pet(BaseModel): name: str type: PetType my_dog: Pet = Pet(name='Lucky', type=PetType.DOG) # This works resp: Response = requests.post('https://postman-echo.com/post', json=my_dog.json()) print(resp.json()) #This doesn't work resp: Response = requests.post('https://postman-echo.com/post', json=my_dog.dict()) print(resp.json())  That when I send json equals to model's dict(), I get the error: TypeError: Object of type 'PetType' is not JSON serializable  How do I overcome this error and make PetType also serializable? P.S. The above example is short and simple, but I hit a use case where both cases of sending json=my_dog.json()  and json=my_dog.dict()  don't work. This is why I need to solve sending using dict().

Answer: **<---- Addition 2 ----> ** Check types like https://docs.python.org/3/library/enum.html#enum.StrEnum and https://docs.python.org/3.12/library/enum.html#enum.IntEnum Instead of MyEnum(str, Enum) use MyEnum(StrENum) **<---- Addition ----> ** Look for Pydantic's parameter "use_enum_values" in Pydantic Model Config use_enum_values whether to populate models with the value property of enums, rather than the raw enum. This may be useful if you want to serialise model.dict() later (default: False) It looks like setting this value to True will do the same as the below solution.  Turns out that this is a behavior of ENum, which is discussed here: https://github.com/samuelcolvin/pydantic/issues/2278 The way you should define the enum is using class PetType(str, Enum):  instead of class PetType(Enum):  For integers this Python's Enum library provides the type IntEnum: https://docs.python.org/3.10/library/enum.html#enum.IntEnum which is basically class IntEnum(int, Enum): pass  If you look at the above Enum documentation you will find that a type like StrEnum doesn't exist but following the example for PetType you can define it easily. I am attaching the working code below from enum import Enum import requests from pydantic import BaseModel from requests import Response class PetType(str, Enum): DOG: str = 'dog' CAT: str = 'cat' class Pet(BaseModel): name: str type: PetType my_dog: Pet = Pet(name='Lucky', type=PetType.DOG) # This works resp: Response = requests.post('https://postman-echo.com/post', json=my_dog.json()) print(resp.json()) # Now this also works resp: Response = requests.post('https://postman-echo.com/post', json=my_dog.dict()) print(resp.json())
Question: How do you convert a column of dates of the form "2020-06-30 15:20:13.078196+00:00" to datetime in pandas? This is what I have done:  pd.concat([df, df.date_string.apply(lambda s: pd.Series({'date':datetime.strptime(s, '%Y-%m-%dT%H:%M:%S.%f%z')}))], axis=1) pd.concat([df, df.file_created.apply(lambda s: pd.Series({'date':datetime.strptime(s, '%Y-%m-%dT%H:%M:%S.%f.%z')}))], axis=1) pd.concat([df, df.file_created.apply(lambda s: pd.Series({'date':datetime.strptime(s, '%Y-%m-%dT%H:%M:%S.%f:%z')}))], axis=1)  I get the error - time data '2020-06-30 15:20:13.078196+00:00' does not match format in all cases. Any help is appreciated.

Answer: +00:00 is a UTC offset of zero hours, thus can be interpreted as UTC. The easiest thing to do is let pd.to_datetime auto-infer the format. That works very well for standard formats like this (ISO 8601): import pandas as pd dti = pd.to_datetime(["2020-06-30 15:20:13.078196+00:00"]) print(dti) # DatetimeIndex(['2020-06-30 15:20:13.078196+00:00'], dtype='datetime64[ns, UTC]', freq=None)  Notes  pandas v2 allows you to set format="ISO8601" to specify that your input is in that format. It even allows parsing a mix of ISO8601 compatible strings: example. pd.to_datetime also works very well for mixed formats: example. In pandas v2, you can set format="mixed" in such cases.
Question: I was using PyDev plugin in eclipse for developing python. But now I switched to NetBeans IDE 12.6 and I searched google for finding python plugins for NetBeans. I found a plugin called nbpython. But it is for NetBeans 8.1 and I am using NetBeans 12.6. So is there any plugin for NetBeans IDE 12.6 for developing Python Projects. Or does nbpython work in my version?

Answer: The new plugin for python is netbeansPython: https://plugins.netbeans.apache.org/catalogue/?id=89 https://github.com/albilu/netbeansPython