eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a Node class holding RGB data in both hex and HSV form. I'll be using this to sort colors in various ways and would prefer the HSV tuple to remain in float form for comparisons instead of converting from a string for every use. Is there a way to specify to the dataclass field that it should format the value in a specific way similar to default values with the default_factory, i.e. a repr_factory? def RGB2HSV(r, g, b): '''Returns HSV values in the range H = [0, 360], S = [0, 100], V = [0, 100]''' r, g, b = r / 255, g / 255, b / 255 maxRGB = max(r, g, b) minRGB = min(r, g, b) delta = maxRGB - minRGB V = maxRGB if V == 0: return 0, 0, V S = delta / V * 100 if S == 0: return 0, S, V * 100 if V == r: H = (g - b) / delta elif V == g: H = 2 + (b - r) / delta else: H = 4 + (r - g) / delta H *= 60 if H < 0: H += 360 return H, S, V * 100 @dataclass class Node: r: int = field(repr=False) g: int = field(repr=False) b: int = field(repr=False) hex: tuple[int, int, int] = field(init=False) hsv: tuple[float, float, float] = field(init=False) def __post_init__(self): self.hex = self.r, self.g, self.b # Generating random r, g, b numbers self.hsv = RGB2HSV(self.hex) # Converts the r, g, b to a tuple of floats  While I'm working out the different sorts, I'm printing out the Nodes and seeing 10 unnecessary digits of a float is distracting. As far as I can think of, would I just be better off implementing my own __repr__ for the class instead of relying on the dataclass generated one? The reason I'm looking at the __repr__ value is because it's automatically generated by the dataclass and can make distinguishing between nearly identical colors easier than just looking at the visual output. It'll be easier to find out what to change or do next if I know what the actual numbers a color are. A portion of the end of the output: Node(hex=(238, 0, 0), hsv=(0.0, 100.0, 93.33333333333333)) Node(hex=(238, 17, 0), hsv=(4.285714285714286, 100.0, 93.33333333333333)) Node(hex=(238, 34, 0), hsv=(8.571428571428571, 100.0, 93.33333333333333)) Node(hex=(238, 51, 0), hsv=(12.857142857142858, 100.0, 93.33333333333333)) Node(hex=(255, 0, 0), hsv=(0.0, 100.0, 100.0)) Node(hex=(255, 17, 0), hsv=(4.0, 100.0, 100.0)) Node(hex=(255, 34, 0), hsv=(8.0, 100.0, 100.0)) Node(hex=(255, 51, 0), hsv=(12.0, 100.0, 100.0))  Basically, can a format be specified to a dataclass field, similar to how a function can be specified to default_factory, in order for the generated __repr__ to format the field for me so I don't have to write my own? ... hsv: tuple[float, float, float] = field(init=False, repr_factory=lambda x: "{:.3f"}.format(x) for x in self.hsv) ...  Node(hex=(238, 51, 0), hsv=(12.857, 100.000, 93.333)) 

Answer: The dataclasses library currently does not support formatting fields like that. The code generated in the default __repr__ for each included field is always in the formf'field={self.field!r}'. You will have to write your own __repr__.
Question: This is my data: z=[[2.021127032949411, 2.405934869144868, 6.005238252515181, 0.43308358365566557, 6.80624302463342, 1.4243920241458794], [6.754588097147502, 17.66441844606136, 17.66863225189955, 4.796490376261003, 4.100119672126023, 2.6461740133188454], [30.522227304933793, 25.244026806049867, 44.77345477001106, 24.58566233495368, 5.070470289616061, 0.9441603389397017], [2.154134557312937, 4.310863690800093, 6.1213216109229505, 3.1274613380516687, 2.5391663573164514, 0.3578307481864878], [19.520038969668185, 10.092300407536902, 1.980581522863168, 2.792253899319521, 0.7083651529637687, 1.7654249187194606]] x=['0.0', '6.784919041781837', '13.569838083563674', '20.354757125345515', '27.139676167127348', '33.92459520890919'] y= [0.0, 1306.1224489795918, 2612.2448979591836, 3918.3673469387754, 5224.489795918367]  My code looks like this: fig = go.Figure(data=go.Heatmap( z=z, x=x, y=y, colorscale='Viridis')) fig.update_layout( xaxis = dict( tickmode = 'linear', tick0 = 0, dtick = 20, ), font=dict(size=18, color="black")) plotly.offline.plot(fig)  And the result is  How can I set nice x axis ticks, for example [0, 5, 10...] with no decimal points? Looks like my attempt above with tick0 and dtick is not helpful.

Answer: Well, hello o/ Follow the code below and adapt/update it how you want ;) import plotly.graph_objects as go fig = go.Figure(data=go.Heatmap( z=z, x=x, y=y, colorscale='Viridis')) fig.update_layout( xaxis = dict( tickmode='array', #change 1 tickvals = x, #change 2 ticktext = [0,5,10,15,20,25], #change 3 ), font=dict(size=18, color="black")) fig.show()  Result (click to zoom in and check):
Question: I am trying to convert a SQL stored procedure to databricks notebook. In the stored procedure below 2 statements are to be implemented. Here the tables 1 and 2 are delta lake tables in databricks cluster. I want to use a python variable in place of max_date in SQL query. How to do it? %sql DELETE FROM table1 WHERE Date = max_date; INSERT INTO table1 SELECT * FROM table2 WHERE Date = max_date;  

Answer: If you are going to run it cell by cell then you can use databricks widgets like First cell x=str(datetime.date.today()) dbutils.widgets.text("max_date",x)  Second cell %sql select getArgument("max_date") AS max_date  will give you max_date 2022-06-04  but as mentioned here it does not work when run all is used and ideal way will be to create separate language based notebook and pass variables using %run Other way is to use spark conf like below First set a value for the conf [Note-the conf name should have .(dot) in it] max_date2=str(datetime.date.today()) spark.conf.set("abc.max_dt2", max_date2)  Next try selecting value like below %sql select "${abc.max_dt2}" as max_date  It should give same value as above
Question: I have a array, I want to pick first 2 or range, skip the next 2, pick the next 2 and continue this until the end of the list list = [2, 4, 6, 7, 9,10, 13, 11, 12,2] results_wanted = [2,4,9,10,12,2] # note how it skipping 2. 2 is used here as and example  Is there way to achieve this in python?

Answer: Taking n number of elements and skipping the next n. l = [2, 4, 6, 7, 9, 10, 13, 11, 12, 2] n = 2 wanted = [x for i in range(0, len(l), n + n) for x in l[i: i + n]] ### Output : [2, 4, 9, 10, 12, 2]
Question: I am trying to get my head around this one. I am quite new to python and even more to image filters and manipulation. I'd like tu use OpenCV to apply this function :  def gaussianFilter(img, kernel_size=(3,3), standard_dev=1): return cv2.GaussianBlur(img, ksize=kernel_size, sigmaX=standard_dev, sigmaY=standard_dev)  Later on in my code I'd want to apply a DoG. According to the definitions I've found, I could have done the following : diff1 = gaussianFilter(img, standard_dev=1) diff2 = gaussianFilter(img, standard_dev=10) res = cv2.subtract(diff2,diff1)  Which should equal to using the following (from skimage.filters): def differenceOfGaussianFilter(img,low_sigma=1, high_sigma=10): return difference_of_gaussians(img, low_sigma=low_sigma, high_sigma=high_sigma)  Obviously the results aren't equal. Even more, changing the parameters in gaussianFilter() doesn't change the output value much... while changing them in differenceOfGaussianFilter() render very different pictures. My aim would be to write a simple function for differenceOfGaussianFilter() but which uses the difference between 2 gaussianFilter() calls. here is an example of the 2 differents results (left is gaussianFilter(10) - gaussianFilter(1) // right is differenceOfGaussianFilter(1,10) )  Any idea on how I could do that ? and why my results aren't similar ? thanks :)

Answer: Kernel size should be proportional to sigma. You didn't adjust that kernel size in these calls, so your sigma=10 lowpass amounts to a 3x3 box blur... which has a sigma of around 1-2, but isn't a gaussian anymore. Rule of thumb: kernel radius ~= 3*sigma. Then you get +- three sigmas of gaussian distribution, which is 99.7% of its probability mass. Best not give an explicit kernel size, but let cv.GaussianBlur calculate that on its own. Pass (0,0) for the size.
Question: I have list of dictionaries and list of integers x = [ { "name": "tom", "job": "judge" }, { "name":"bob", "job": "policeman" } ] y = [1000, 2200]  I want to zip them and add y elements to dictionaries as "payroll": y_element The desired output would be: [ { "name": "tom", "job": "judge", "payroll": 1000 }, { "name":"bob", "job": "policeman", "payroll": 2200 } ]  I actually achieved that by: z = zip(x, y) for i in z: i[0]["payroll"] = i[1] z = [i[0] for i in z]  But I was wondering whether it could be done in dict comprehension in list comprehension. This is what I tried so far: z = [{k: v, "value": o} for d, o in z for k, v in d.items()]  Obviously it is wrong because the output is: {'name': 'bob', 'job': 'policeman', 'value': 2} 

Answer: You can merge the dict with the required data using ** here. [{**d, 'payroll':i} for d, i in zip(x, y)] # [{'name': 'tom', 'job': 'judge', 'payroll': 1000}, # {'name': 'bob', 'job': 'policeman', 'payroll': 2200}]
Question: For two arrays, say, a = np.array([1,2,3,4]) and b = np.array([5,6]), is there a way, if any, to obtain a 2d array of the following form without looping: [[5 6 1 2 3 4] [1 5 6 2 3 4] [1 2 5 6 3 4] [1 2 3 5 6 4] [1 2 3 4 5 6]]  i.e. to insert b in all possible places of a. And if loops are unavoidable, how to do it the most computationally efficient way (a can be long, the length of b is irrelevant)? Example of how it can be done using loops is trivial: a = np.array([1,2,3,4]) b = np.array([5,6]) rows = len(a) + 1 cols = len(a) + len(b) res = np.empty([rows, cols]) for i in range(rows): res[i, i:len(b)+i] = b res[i, len(b)+i:] = a[i:] res[i, 0:i] = a[0:i] print(rows.astype(int)) [[5 6 1 2 3 4] [1 5 6 2 3 4] [1 2 5 6 3 4] [1 2 3 5 6 4] [1 2 3 4 5 6]] 

Answer: Consider using numba acceleration. This happens to be what numba is best at. For your example, it can speed up nearly 6 times: from timeit import timeit import numpy as np from numba import njit a = np.arange(1, 5) b = np.array([5, 6]) def fill(a, b): rows = a.size + 1 cols = a.size + b.size res = np.empty((rows, cols)) for i in range(rows): res[i, i:b.size + i] = b res[i, b.size + i:] = a[i:] res[i, :i] = a[:i] return res if __name__ == '__main__': print('before:', timeit(lambda: fill(a, b))) fill = njit(fill) print('after:', timeit(lambda: fill(a, b)))  Output: before: 9.488150399993174 after: 1.6149254000047222
Question: I want to convert generators to reusable iterables. For example, consider generator: def myrange(n): for i in range(n): yield i  It generates one-use iterator: x = myrange(3) print(list(x)) # [0, 1, 2] print(list(x)) # []  I want to add a decorator to the definition of myrange such that it produces reusable iterable (like the usual range) instead of one-use iterator: x = myrange(3) print(list(x)) # [0, 1, 2] print(list(x)) # [0, 1, 2] 

Answer: This should work: def mk_reusable(f): """ Makes a reusable iterable out of generator by remembering its arguments """ class MyIterable: def __init__(self, *args, **kwargs): self._args = args self._kwargs = kwargs def __iter__(self): yield from f(*self._args, **self._kwargs) return MyIterable # TEST @mk_reusable def myrange(n): for i in range(n): yield i x = myrange(3) assert list(x) == [0, 1, 2] assert list(x) == [0, 1, 2] # can consume it twice
Question: The output of: class Dog(): def get_class(): return __class__ class Cat(): def get_class(): return __class__ print(Dog.get_class()) print(Cat.get_class())  is: <class '__main__.Dog'> <class '__main__.Cat'>  I want to DRY up my code with a subclass. But the output of: class BaseClass(): def get_class(): return __class__ class Dog(BaseClass): pass class Cat(BaseClass): pass print(Dog.get_class()) print(Cat.get_class())  is <class '__main__.BaseClass'> <class '__main__.BaseClass'>  How do I change the code in the second case to obtain the same output as the first case?

Answer: you are almost there : class BaseClass: @classmethod def get_class(cls): return cls class Dog(BaseClass): pass class Cat(BaseClass): pass print(Dog.get_class()) print(Cat.get_class()) <class '__main__.Dog'> <class '__main__.Cat'>
Question: I have an array of datetimes that I need to convert to a list of datetimes. My array looks like this: import numpy as np my_array = np.array(['2017-06-28T22:47:51.213500000', '2017-06-28T22:48:37.570900000', '2017-06-28T22:49:46.736800000', '2017-06-28T22:50:41.866800000', '2017-06-28T22:51:17.024100000', '2017-06-28T22:51:24.038300000'], dtype='datetime64[ns]') my_list = my_array.tolist()  I need a list of datetime values, but when I do my_array.tolist(), I get a list of numerical time stamps: [1498690071213500000, 1498690117570900000, 1498690186736800000, 1498690241866800000, 1498690277024100000, 1498690284038300000]  My question is how do I preserve the datetime format when going from an array to a list, or how do I convert the list of time stamps to a list datetime values?

Answer: Explicitly casting the numpy.ndarray as a native Python list will preserve the contents as numpy.datetime64 objects: >>> list(my_array) [numpy.datetime64('2017-06-28T22:47:51.213500000'), numpy.datetime64('2017-06-28T22:48:37.570900000'), numpy.datetime64('2017-06-28T22:49:46.736800000'), numpy.datetime64('2017-06-28T22:50:41.866800000'), numpy.datetime64('2017-06-28T22:51:17.024100000'), numpy.datetime64('2017-06-28T22:51:24.038300000')]  However, if you wanted to go back from an integer timestamp to a numpy.datetime64 object, the number given here by numpy.ndarray.tolist is given in nanosecond format, so you could also use a list comprehension like the following: >>> [np.datetime64(x, "ns") for x in my_list] [numpy.datetime64('2017-06-28T22:47:51.213500000'), numpy.datetime64('2017-06-28T22:48:37.570900000'), numpy.datetime64('2017-06-28T22:49:46.736800000'), numpy.datetime64('2017-06-28T22:50:41.866800000'), numpy.datetime64('2017-06-28T22:51:17.024100000'), numpy.datetime64('2017-06-28T22:51:24.038300000')]  And if you want the final result as a Python datetime.datetime object instead of a numpy.datetime64 object, you can use a method like this (adjusted as needed for locality): >>> from datetime import datetime >>> list(map(datetime.utcfromtimestamp, my_array.astype(np.uint64) / 1e9)) [datetime.datetime(2017, 6, 28, 22, 47, 51, 213500), datetime.datetime(2017, 6, 28, 22, 48, 37, 570900), datetime.datetime(2017, 6, 28, 22, 49, 46, 736800), datetime.datetime(2017, 6, 28, 22, 50, 41, 866800), datetime.datetime(2017, 6, 28, 22, 51, 17, 24100), datetime.datetime(2017, 6, 28, 22, 51, 24, 38300)]  Edit: Warren Weckesser's answer provides a more straightforward approach to go from a numpy.datetime64[ns] array to a list of Python datetime.datetime objects than is described here.
Question: I just made a module in python but I don't want people to do this: import mymodule mymodule.  and this then shows all methods and variables I added to the module. I just want them to see specified ones because I have many additional ones that are only for internal use.

Answer: If this is your module mymodule.py: def expose_this(): print('yes please') def but_not_this(): print('definitely not')  Importing it directly like this: import mymodule  Gives a user access to mymodule.expose_this() and mymodule.but_not_this(), there's no way to change that, although you could call but_not_this something like _but_not_this instead and many editors would at least warn a user that they are not supposed to access that. However, the right way to do it would be to create a package. Put your module in a separate folder (called mymodule), and add an __init__.py with this: from .mymodule import expose_this  Now, if someone imports your package using the same import statement as before, they only have access to mymodule.expose_this() import mymodule mymodule.expose_this() # this works mymodule.but_not_this() # this causes an error  There's a lot more to say about creating packages and how to add content, but there's good documentation and tutorials for that out there. Note: if someone knows the internal structure of your module, they can still access but_not_this() with this: import mymodule mymodule.mymodule.but_not_this()  But they'd have to really want to - making it impossible is hard and there's really no point, since someone will be able to get at your code anyway, if they need to. But if you want to make the intent clear, you could rename mymodule.py to _mymodule.py and prefix the functions you don't want exposed with a _ as well - this helps editors to reinforce your intentions with the user.
Question: I am trying to connect my gmail to python, but show me this error: I already checked my password, any idea what can be? b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)' Traceback (most recent call last): File "/Users/myuser/Documents/migrations/untitled3.py", line 29, in read_email_from_gmail mail.login(FROM_EMAIL,FROM_PWD) File "/Users/myuser/opt/anaconda3/lib/python3.9/imaplib.py", line 612, in login raise self.error(dat[-1]) imaplib.IMAP4.error: b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'  Here my code: Also i want to know which port I can use? import smtplib import time import imaplib import email import traceback ORG_EMAIL = "@gmail.com" FROM_EMAIL = "myemail" + ORG_EMAIL FROM_PWD = "mypassword" SMTP_SERVER = "smtp.gmail.com" SMTP_PORT = ?? def read_email_from_gmail(): try: mail = imaplib.IMAP4_SSL(SMTP_SERVER) mail.login(FROM_EMAIL,FROM_PWD) mail.select('inbox') data = mail.search(None, 'ALL') mail_ids = data[1] id_list = mail_ids[0].split() first_email_id = int(id_list[0]) latest_email_id = int(id_list[-1]) for i in range(latest_email_id,first_email_id, -1): data = mail.fetch(str(i), '(RFC822)' ) for response_part in data: arr = response_part[0] if isinstance(arr, tuple): msg = email.message_from_string(str(arr[1],'utf-8')) email_subject = msg['subject'] email_from = msg['from'] print('From : ' + email_from + '\n') print('Subject : ' + email_subject + '\n') except Exception as e: traceback.print_exc() print(str(e)) read_email_from_gmail()  My main goal is be able to get the CSV file from each email, but for now I just want to read messages.

Answer:  You need to enable 'Less secure apps' in your Gmail account if you're going to check it this way. For that reason, it would be better to use the Gmail API.  SMTP port is not set - ensure you are using the correct port (993)
Question: I am trying to send email using python. My code was working fine before Google disabled 'less secure apps'. My email address and password are both correct. server = smtplib.SMTP_SSL("smtp.gmail.com", 465) serverEmail = "EMAILADDRESS" serverPw = "QWERTY" server.login(serverEmail, serverPw) subject = "Rejection" body = "Hi! You've been unfortunately declined access to our system." message = f'Subject: {subject}\n\n{body}' server.sendmail("EMAILADDRESS", doctorEmail['email'], message) server.quit()  I get this error now:  smtplib.SMTPAuthenticationError: (535, b'5.7.8 Username and Password not accepted.  I get this error when i use server.starttls():  smtplib.SMTPNotSupportedError: STARTTLS extension not supported by server. 

Answer: This is working for me. You need to generate an app password for this. See https://support.google.com/accounts/answer/185833?hl=en import smtplib as smtp connection = smtp.SMTP_SSL('smtp.gmail.com', 465) email_addr = 'my_email@gmail.com' email_passwd = 'app_password_generated_in_Google_Account_Settings' connection.login(email_addr, email_passwd) connection.sendmail(from_addr=email_addr, to_addrs='recipient@something.com', msg="Sent from my IDE. Hehe") connection.close()  For some reason, all of my emails are ending up in SPAM folder of the recipient account though.
Question: I tried to use Shap (Tree Explainer) for sklearn.ensemble._stacking.StackingClassifier explainer = shap.TreeExplainer(clf) shap_values = explainer.shap_values(x) shap.initjs() return shap.force_plot(explainer.expected_value[1], shap_values[1], x)  But I got an error: Model type not yet supported by TreeExplainer: <class 'sklearn.ensemble._stacking.StackingClassifier'> How can I use shap force_plot for sklearn StackingClassifier? Thank you.

Answer: TreeExplainer only works on tree-based models themselves, not on pipelines or metamodels that end with a tree-based model. If you want interpretability in terms of your original features, you will need to use the base Explainer class (or equivalently, the KernelExplainer class). Unfortunately, this will be approximate and more computationally expensive.
Question: How to filter out NaN in pytdantic float validation? from pydantic import BaseModel class MySchema(BaseModel): float_value: float 

Answer: You can use confloat and set either the higher limit to infinity or the lower limit to minus infinity. As all numeric comparisons with NaN return False, that will make pydantic reject NaN, while leaving all other behaviour identical (including parsing, conversion from int to float, ...). from pydantic import BaseModel, confloat class MySchema(BaseModel): float_value: confloat(ge=-float('inf')) # or: # float_value: confloat(le=float('inf'))  Note: you could additionally exclude infinity values by using the gt and lt arguments of confloat instead of ge and le. Testing: m = MySchema(float_value=float('nan'))  Output: pydantic.error_wrappers.ValidationError: 1 validation error for MySchema float_value ensure this value is greater than or equal to -inf (type=value_error.number.not_ge; limit_value=-inf)
Question: I'm currently trying to generate python code from a proto file. My proto file looks like this: syntax = "proto3"; package display; message Hello { uint32 version = 1; uint32 value = 2; int32 id = 3; }  I used this protoc command to generate the python code: protoc -I="." --python_out="." test.proto  And here is the resulting python file: # -*- coding: utf-8 -*- # Generated by the protocol buffer compiler. DO NOT EDIT! # source: test.proto """Generated protocol buffer code.""" from google.protobuf.internal import builder as _builder from google.protobuf import descriptor as _descriptor from google.protobuf import descriptor_pool as _descriptor_pool from google.protobuf import symbol_database as _symbol_database # @@protoc_insertion_point(imports) _sym_db = _symbol_database.Default() DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\ntest.proto\x12\x07\x64isplay\"3\n\x05Hello\x12\x0f\n\x07version\x18\x01 \x01(\r\x12\r\n\x05value\x18\x02 \x01(\r\x12\n\n\x02id\x18\x03 \x01(\x05\x62\x06proto3') _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals()) _builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'test_pb2', globals()) if _descriptor._USE_C_DESCRIPTORS == False: DESCRIPTOR._options = None _HELLO._serialized_start=23 _HELLO._serialized_end=74 # @@protoc_insertion_point(module_scope)  It doesn't look at all like the documentation from Google on this page. Why isn't the metaclass generated? I'm using Python 3.9 with the latest version of the protobuf package and last version of protoc.

Answer: add --grpc_python_out="." to the protoc command. this will generate an additional script with the required classes