eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: After read excel file with pandas, gets the follow warning: key code:  pd_obj = pd.read_excel("flie.xls", dtype=str, usecols=usecols, skiprows=3) for idx, row in pd_obj.iterrows(): json_tmpl = copy.deepcopy(self.details) json_tmpl["nameInBank"] = row["nameInBank"] json_tmpl["totalBala"] = row["totalBala"].replace(",", '') # parse pdf file status = self._get_banksplip_json(json_tmpl["bankReceipts"], row) json_buf.append(copy.deepcopy(json_tmpl))  warning info : WARNING *** file size (48130) not 512 + multiple of sector size (512) WARNING *** file size (44546) not 512 + multiple of sector size (512) 

Answer: This appears to be a normal warning from the underlying XLRD library, and it seems safe to ignore. A pandas issue (#16620) was opened and closed without a conclusive resolution. However, the discussion did provide an alternative that would allow you to suppress the warnings: from os import devnull import pandas as pd import xlrd wb = xlrd.open_workbook('file.xls', logfile=open(devnull, 'w')) pd_obj = pd.read_excel(wb, dtype=str, usecols=usecols, skiprows=3, engine='xlrd')  You can read a more detailed analysis of the actual cause of the error on the forum here: https://groups.google.com/forum/m/#!topic/python-excel/6Lue-1mTPSM Moral of the story: whenever you get a warning you aren't sure about, you should search for the keywords that appear (discard any specific parts like file sizes or local paths). This answer is based on the first two results to show up on Google.
Question: I have the following code on a.py: class Tags(enum.Flag): NONE = 0 A = enum.auto() B = enum.auto() C = enum.auto() # Allow using tags.A instead of tags.Tags.A globals().update(Tags.__members__)  But when I use it on other files, mypy (rightfully) fails to identify the attributes: import tags print(tags.A) # Module has no attribute "A"  Is there any possible way to bypass this in Python 3.6? Known solutions (which are not good enough for my case):  Use # type: ignore every time you use tags.A Use tags.Tags.A Use __getitem__ in the module level (Works only from Python 3.7) 

Answer: Personally, what I would do is modify my imports to do: from tags import Tags  ..which would let me refer to the enum items by doing just Tags.A everywhere with minimal fuss.  But if you really want to continue referring to these items from the module namespace instead, one approach is to define a __getattr__ function within tags.py: class Tags(enum.Flag): NONE = 0 A = enum.auto() B = enum.auto() C = enum.auto() def __getattr__(name: str) -> Tags: return Tags[name]  At runtime, Python will try calling this function if you try accessing some attribute that wasn't directly defined in the tags module object. Mypy understands this convention and assumes that if __getattr__ is defined, the module is "incomplete". So, it'll use the return type as the type for any missing attributes you try accessing. You may still want to do globals().update(Tags.__members__) mostly as a performance optimization though, to skip having to actually call __getattr__ function at runtime. This strategy only really works if tags.py only contains one enum though -- otherwise, you'd need to make the return type something like Union[Tags, MyOtherEnum] (which is unwieldy) or even just Any (which loses the benefits of using a type checker). This strategy also means mypy won't be able to do more sophisticated type inference and narrowing by taking into account the actual enum value. This is mostly only relevant if you're using Literal enums though.  If these are concerns, you might have to resort to a more brute force approach, like so: from typing import TYPE_CHECKING class Tags(enum.Flag): NONE = 0 A = enum.auto() B = enum.auto() C = enum.auto() globals().update(Tags.__members__) if TYPE_CHECKING: NONE = Tags.NONE A = Tags.A B = Tags.B C = Tags.C  The TYPE_CHECKING constant is always false at runtime, but is considered to be always true by type checkers. But if you're going to go to the expense of directly telling mypy about each of these variants, you might as well just skip trying to automatically update the globals and do this instead: class Tags(enum.Flag): NONE = 0 A = enum.auto() B = enum.auto() C = enum.auto() NONE = Tags.NONE A = Tags.A B = Tags.B C = Tags.C  Obviously, having to repeat yourself twice like this is pretty suboptimal, but I don't think there's an easy way around it. You could perhaps mitigate this by creating a script that auto-generates tags.py for you, but that's also fairly suboptimal, just for different reasons.
Question: I have the data in dataframes like below. I want to split the item into same number of rows >>> df idx a 0 3 1 5 2 4  from above dataframe, I want the below as  >>> df idx a 0 1 1 2 2 3 3 1 4 2 5 3 6 4 7 5 8 1 9 2 10 3 11 4  I tried several ways, but no success. 

Answer: A Fun way  df.a.map(range).explode()+1 # may add reset_index(), however, I think keep the original index is good, and help us convert back. Out[158]: idx 0 1 0 2 0 3 1 1 1 2 1 3 1 4 1 5 2 1 2 2 2 3 2 4 Name: a, dtype: object
Question: If we have a string of alphabetical characters and some dashes, and we want to count the number of dashes between any two alphabetic characters in this string. what is the easiest way to do this? Example: Input: a--bc---d-k output: 2031 This means that there are 2 dashes between a and b, 0 dash between b and c, 3 dashes between c and d and 1 dash between d and k what is a good way to find this output list in python?

Answer: Solution with regex: import re x = 'a--bc---d-k' results = [ len(m) for m in re.findall('(?<=[a-z])-*(?=[a-z])', x) ] print(results) print(''.join(str(r) for r in results))  output: [2, 0, 3, 1] 2031   Solution with brute force loop logic: x = 'a--bc---d-k' count = 0 results = [] for c in x: if c == '-': count += 1 else: results.append(count) count = 0 results = results[1:] # cut off first length print(results)  output: [2, 0, 3, 1]
Question: Using Python's print() (or to the best of my knowledge, any other console output generating) function in loop structures and running the code via reticulate in R, the output is only printed after the execution finished. For example, take the following loop which goes to sleep for 1.5 seconds after each iteration; the run numbers are all printed in one go after the loop is through. The same applies when saving the Python code to a separate .py file and then running reticulate::py_run_file(). library(reticulate) py_run_string(" import time for i in range(5): print(str(i)) time.sleep(1.5) # sleep for 1.5 sec ")  Does anyone know where this behavior comes from and, if possible, how to circumvent it?

Answer: Apparently, you need to force pyhton to export the standard output at times. You can do this by adding sys.stdout.flush() to your code: library(reticulate) py_run_string(" import time import sys for i in range(5): print(str(i)) time.sleep(1.5) # sleep for 1.5 sec sys.stdout.flush() ")  see over here described with nohup
Question: I'm using the Python typing module throughout my project, and I was wondering if there was a way to specify that a given object must be of two different types, at once. This most obviously arises when you have specified two protocols, and expect a single object to fulfil both: class ReturnsNumbers(Protocol): def get_number(self) -> Int: pass class ReturnsLetters(Protocol): def get_letter(self) -> str: pass def get_number_and_letter(x: <what should this be?>) -> None: print(x.get_number(), x.get_letter())  Thanks in advance!

Answer: Create a new type that inherits from all types you wish to combine, as well as Protocol. class ReturnsNumbersAndLetters(ReturnsNumbers, ReturnsLetters, Protocol): pass def get_number_and_letter(x: ReturnsNumbersAndLetters) -> None: print(x.get_number(), x.get_letter())
Question: Why does Mypy complain that it requires a type annotation for a list comprehension variable, when it is impossible to annotate such a variable using MyPy? Specifically, how can I resolve the following error: from enum import EnumMeta def spam( y: EnumMeta ): return [[x.value] for x in y] 🠜 Mypy: Need type annotation for 'x'  cast doesn't work: return [[cast(Enum, x).value] for x in y] 🠜 Mypy: Need type annotation for 'x'  Even though Mypy doesn't support annotations (x:Enum) in such a case I see the usage of the variable can be annotated using cast (see this post). However, cast(Enum, x) doesn't stop Mypy complaining that the variable isn't annotated in the first place. #type: doesn't work: return [[x.value] for x in y] # type: Enum 🠜 Mypy: Misplaced type annotation  I also see that a for loop variable can be annotated using a comment, # type: (see this post). However, # type: Enum doesn't work for list comprehension's for.

Answer: In a list comprehension, the iterable must be cast instead of the elements. from typing import Iterable, cast from enum import EnumMeta, Enum def spam(y: EnumMeta): return [[x.value] for x in cast(Iterable[Enum], y)]  This allows mypy to infer the type of x as well. In addition, at runtime it performs only 1 cast instead of n casts. If spam can digest any iterable that produces enums, it is easier to type hint this directly. from typing import Iterable from enum import Enum def spam(y: Iterable[Enum]): return [[x.value] for x in y]
Question: I have a Spark RDD of over 6 billion rows of data that I want to use to train a deep learning model, using train_on_batch. I can't fit all the rows into memory so I would like to get 10K or so at a time to batch into chunks of 64 or 128 (depending on model size). I am currently using rdd.sample() but I don't think that guarantees I will get all rows. Is there a better method to partition the data to make it more manageable so that I can write a generator function for getting batches? My code is below: data_df = spark.read.parquet(PARQUET_FILE) print(f'RDD Count: {data_df.count()}') # 6B+ data_sample = data_df.sample(True, 0.0000015).take(6400) sample_df = data_sample.toPandas() def get_batch(): for row in sample_df.itertuples(): # TODO: put together a batch size of BATCH_SIZE yield row for i in range(10): print(next(get_batch())) 

Answer: I don't believe spark let's you offset or paginate your data. But you can add an index and then paginate over that, First:  from pyspark.sql.functions import lit data_df = spark.read.parquet(PARQUET_FILE) count = data_df.count() chunk_size = 10000 # Just adding a column for the ids df_new_schema = data_df.withColumn('pres_id', lit(1)) # Adding the ids to the rdd rdd_with_index = data_df.rdd.zipWithIndex().map(lambda (row,rowId): (list(row) + [rowId+1])) # Creating a dataframe with index df_with_index = spark.createDataFrame(rdd_with_index,schema=df_new_schema.schema) # Iterating into the chunks for page_num in range(0, count+1, chunk_size): initial_page = page_num*chunk_size final_page = initial_page + chunk_size where_query = ('pres_id > {0} and pres_id <= {1}').format(initial_page,final_page) chunk_df = df_with_index.where(where_query).toPandas() train_on_batch(chunk_df) # <== Your function here  This is not optimal it will badly leverage spark because of the use of a pandas dataframe but will solve your problem. Don't forget to drop the id if this affects your function.
Question:  This is a truck container image but from the top view. First, I need to find the rectangle and know each corner position. The goal is to know the dimension of the container.

Answer: Here's a simple approach:  Obtain binary image. Load image, convert to grayscale, Gaussian blur, then Otsu's threshold.  Find distorted bounding rectangle contour and corners. We find contours then filter using contour area to isolate the rectangular contour. Next we find the distorted bounding rectangle with cv2.minAreaRect() and the corners with cv2.boxPoints()    Detected bounding rectangle -> Mask -> Detected corners    Corner points (188, 351) (47, 348) (194, 32) (53, 29)  Code import cv2 import numpy as np # Load image, grayscale, blur, Otsu's threshold image = cv2.imread('1.png') mask = np.zeros(image.shape[:2], dtype=np.uint8) gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) blur = cv2.GaussianBlur(gray, (5,5), 0) thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1] # Find distorted bounding rect cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) cnts = cnts[0] if len(cnts) == 2 else cnts[1] for c in cnts: area = cv2.contourArea(c) if area > 5000: # Find distorted bounding rect rect = cv2.minAreaRect(c) corners = cv2.boxPoints(rect) corners = np.int0(corners) cv2.fillPoly(mask, [corners], (255,255,255)) # Draw corner points corners = corners.tolist() print(corners) for corner in corners: x, y = corner cv2.circle(image, (x, y), 5, (36,255,12), -1) cv2.imshow('thresh', thresh) cv2.imshow('image', image) cv2.imshow('mask', mask) cv2.waitKey()
Question: Recently,I make an app and upload it to my GitHub release page.I want to make a function to check update to get the latest version(in the release pages). I try to use requests module to crawl my release page to get the latest version.Here it a minimal example of my code: import requests from lxml import etree response = requests.get("https://github.com/v2ray/v2ray-core/releases") html = etree.HTML(response.text) Version = html.xpath("/html/body/div[4]/div/main/div[3]/div/div[2]/div[1]/div/div[2]/div[1]/div/div/a") print(Version)  I think the xpath is right,because I use chrome -> copy -> copy xpath.But it return me a [].it can't find the latest version.

Answer: A direct way is to use GitHub API, it is easy to do and don't need xpath.  The URL should be: https://api.github.com/repos/{owner}/{repo}/releases/latest  So if you want to get the latest release version of the repository. Here is an easy example only using the requests module: import requests response = requests.get("https://api.github.com/repos/v2ray/v2ray-core/releases/latest") print(response.json()["name"])
Question: I am trying to visualize multiple 2d trajectories (x, y) in a 3D scatter plot where the z axis is time. import numpy as np import pandas as pd import plotly.express as px # Sample data: 3 trajectories t = np.linspace(0, 10, 200) df = pd.concat([pd.DataFrame({'x': 900 * (1 + np.cos(t + 5 * i)), 'y': 400 * (1 + np.sin(t)), 't': t, 'id': f'id000{i}'}) for i in [0, 1, 2]]) # 3d scatter plot fig = px.scatter_3d(df, x='x', y='y', z='t', color='id', ) fig.update_traces(marker=dict(size=2)) fig.show()   I have a .png file of a map with size: 2000x1000. The (x, y) coordinates of the trajectories correspond to the pixel locations of the map. I would like to see the image of the map on the "floor" of the 3d scatter plot. I have tried to add the image with this code: from scipy import misc img = misc.imread('images/map_bg.png') fig2 = px.imshow(img) fig.add_trace(fig2.data[0]) fig.show()  But the result is having an independent image in the background as a separate plot:  And I want the image on the "floor" of the scatter plot and moving with the scatter plot, if I rotate/zoom. Here is a mock:  Additional note: There can be any number of trajectories and for my application, it is important that each trajectory is automatically plotted with a different color. I am using plotly.express, but I can use other plotly packages, as long as these requirements are met.

Answer: I've ran into the same situation where I wanted to use an image as a bottom surface in a 3D scatterplot. With help from two posts here and here, I've been able to create the following 3d scatter plot:  I've used plotly go in my example, so the result is a little bit different than the code from the OP. import numpy as np import pandas as pd from PIL import Image import plotly.graph_objects as go from scipy import misc im = misc.face() im_x, im_y, im_layers = im.shape eight_bit_img = Image.fromarray(im).convert('P', palette='WEB', dither=None) dum_img = Image.fromarray(np.ones((3,3,3), dtype='uint8')).convert('P', palette='WEB') idx_to_color = np.array(dum_img.getpalette()).reshape((-1, 3)) colorscale=[[i/255.0, "rgb({}, {}, {})".format(*rgb)] for i, rgb in enumerate(idx_to_color)] # Sample data: 3 trajectories t = np.linspace(0, 10, 200) df = pd.concat([pd.DataFrame({'x': 400 * (1 + np.cos(t + 5 * i)), 'y': 400 * (1 + np.sin(t)), 't': t, 'id': f'id000{i}'}) for i in [0, 1, 2]]) # im = im.swapaxes(0, 1)[:, ::-1] colors=df['t'].to_list() # # 3d scatter plot x = np.linspace(0,im_x, im_x) y = np.linspace(0, im_y, im_y) z = np.zeros(im.shape[:2]) fig = go.Figure() fig.add_trace(go.Scatter3d( x=df['x'], y=df['y'], z=df['t'], marker=dict( color=colors, size=4, ) )) fig.add_trace(go.Surface(x=x, y=y, z=z, surfacecolor=eight_bit_img, cmin=0, cmax=255, colorscale=colorscale, showscale=False, lighting_diffuse=1, lighting_ambient=1, lighting_fresnel=1, lighting_roughness=1, lighting_specular=0.5, )) fig.update_layout( title="My 3D scatter plot", width=800, height=800, scene=dict(xaxis_visible=True, yaxis_visible=True, zaxis_visible=True, xaxis_title="X", yaxis_title="Y", zaxis_title="Z" , )) fig.show()
Question: Suppose I have a dataclass: @dataclass(frozen=True) class Foo: id: str name: str  I want this to be immutable (hence the frozen=True), such that foo.id = bar and foo.name = baz fail. But, I want to be able to strip the id, like so: foo = Foo(id=10, name="spam") foo.strip_id() foo -> Foo(id=None, name="spam")  I have tried a few things, overriding setattr, but nothing worked. Is there an elegant solution to this? (I know I could write a method that returns a new frozen instance that is identical except that that id has been stripped, but that seems a bit hacky, and it would require me to do foo = foo.strip_id(), since foo.strip_id() would not actually change foo) Edit: Although some commenters seem to disagree, I think there is a legitimate distinction between 'fully mutable, do what you want with it', and 'immutable, except in this particular, tightly controlled way'

Answer: Well, you can do it by directly modifying the __dict__ member of the instance modifying the attribute using object.__setattr__(...)1, but why??? Asking specifically for immutable and then making it mutable is... indecisive. But if you must: from dataclasses import dataclass @dataclass(frozen=True) class Foo: id: str name: str def strip_id(self): object.__setattr__(self, 'id', None) foo=Foo(10, 'bar') >>> foo Foo(id=10, name='bar') >>> foo.strip_id() >>> foo Foo(id=None, name='bar')  Any way of doing this is probably going to seem hacky... because it requires doing things that are fundamentally the opposite of the design. If you're using this as a signal to other programmers that they should not modify the values, the way that is normally done in Python is by prefixing the variable name with a single underscore. If you want to do that, while also making the values accessible, Python has a builtin module called property, where (from the documentation) "typical use is to define a managed attribute": from dataclasses import dataclass @dataclass class Foo: _name: str @property def name(self): return self._name @name.setter def name(self, value): self._name = value @name.deleter def name(self): self._name = None  Then you can use it like this: >>> f=Foo() >>> f.name = "bar" >>> f.name 'bar' >>> f._name 'bar' >>> del f.name >>> f.name >>> f._name  The decorated methods hide the actual value of _name behind name to control how the user interacts with that value. You can use this to apply transformation rules or validation checks to data before it is stored or returned. This doesn't quite accomplish the same thing as using @dataclass(frozen=True), and if you try declaring it as frozen, you'll get an error. Mixing frozen dataclasses with the property decorator is not straightforward and I have not seen a satisfying solution that is concise and intuitive. @Arne posted this answer, and I found this thread on GitHub, but neither approach is very inspiring; if I came across such things in code that I had to maintain, I would not be very happy (but I would be confused, and probably pretty irritated).  1: Modified as per the answer by @Arne, who observed that the internal use of a dictionary as the data container is not guaranteed.
Question: I'm using a very new Python package manager called Poetry.  It creates several files/directories upon creating a new project (environment), but I'm not sure which one I should add to .gitignore for the best practice.  Say I create a new poetry project by doing this:  $ poetry new foo_project $ cd foo_project $ poetry add numpy $ ls  There are: tests (directory) foo_project (also a directory) pyproject.toml (a file that specifies installed packages) poetry.lock (a lock file of installed packages) README.rst (I don't know why README is created but it just shows up.)  I usually add tests/, foo_project/, poetry.lock and README.rst because they seem to be dependent on the machine the project was created. Also, I seem to be able to reproduce the environment only with pyproject.toml so that's another reason I ignored all other files/directories.  However, it's just my hunch and unfortunately, I can't find any official guide what I really should add to .gitignore on the official documentation.  It just bugs me that I don't know for sure what I'm doing. Which ones should I add to my .gitignore?

Answer: Also moved to poetry quite recently. I would say you should not add any of: tests/, foo_project/, poetry.lock or README.rst to your .gitignore. In other words, those files and folders should be in version control. My reasons are as follows: tests/ - your tests should not be machine dependent (unless this is a know limitation of your package) and providing the tests for others is how they test a) the installation has worked and b) any changes they make don't break past features, so a pull request becomes more robust. foo_project/ - this is where your python module goes! All you .py files should be inside this folder if you want poetry to be able to build and publish your package. poetry.lock - see https://python-poetry.org/docs/basic-usage/ where it says:  When Poetry has finished installing, it writes all of the packages and the exact versions of them that it downloaded to the poetry.lock file, locking the project to those specific versions. You should commit the poetry.lock file to your project repo so that all people working on the project are locked to the same versions of dependencies (more below).  README.rst - although this one is perhaps more a personal thing, this is the file that becomes your package readme if you use poetry for publishing your package, e.g. to PyPi. Without it you're package will have an empty readme. I have two readmes one .md (for GitHub) and one .rst (for PyPi). I use the GitHub readme for developers/users and the PyPi for pure users.