eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm building a fairly large JSON dictionary where I specify a few uuids like this: import uuid game['uuid'] = uuid.uuid1()  I'm getting a type error with the following traceback. I'm not sure what the issue is because we can have UUIDs within json objects Traceback (most recent call last): File "/Users/claycrosby/Desktop/coding/projects/gambling/scraper/sbtesting.py", line 182, in <module> game_json = json.dumps(game) File "/opt/miniconda3/envs/ds383/lib/python3.8/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/opt/miniconda3/envs/ds383/lib/python3.8/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/opt/miniconda3/envs/ds383/lib/python3.8/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/opt/miniconda3/envs/ds383/lib/python3.8/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type UUID is not JSON serializable [Finished in 0.5s with exit code 1] [cmd: ['/opt/miniconda3/envs/ds383/bin/python3', '/Users/claycrosby/Desktop/coding/projects/gambling/scraper/sbtesting.py']] [dir: /Users/claycrosby/Desktop/coding/projects/gambling/scraper] [path: /opt/miniconda3/bin:/opt/miniconda3/condabin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/Users/claycrosby/Desktop/coding/programs/pbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin] 

Answer: The uuid.UUID class itself can't be JSON serialized, but the object can be expressed in several JSON compatible formats. From help(uuid.UUID) we can see these options (though bytes would need some more work because they aren't json either).  | bytes the UUID as a 16-byte string (containing the six | integer fields in big-endian byte order) | | bytes_le the UUID as a 16-byte string (with time_low, time_mid, | and time_hi_version in little-endian byte order) | | fields a tuple of the six integer fields of the UUID, | which are also available as six individual attributes | and two derived attributes: | | time_low the first 32 bits of the UUID | time_mid the next 16 bits of the UUID | time_hi_version the next 16 bits of the UUID | clock_seq_hi_variant the next 8 bits of the UUID | clock_seq_low the next 8 bits of the UUID | node the last 48 bits of the UUID | | time the 60-bit timestamp | clock_seq the 14-bit sequence number | | hex the UUID as a 32-character hexadecimal string | | int the UUID as a 128-bit integer | | urn the UUID as a URN as specified in RFC 4122  If your API wants a URN for example, you'd >>> game = {'uuid': uuid.uuid1().urn} >>> game {'uuid': 'urn:uuid:56fabaca-0fe6-11eb-9910-c770eddca9e7'} >>> json.dumps(game) '{"uuid": "urn:uuid:56fabaca-0fe6-11eb-9910-c770eddca9e7"}'
Question: I'm trying to install the csv module in Python 3. I have pip install, and I'm using Pycharm. I've tried downloading it in the terminal using both pip install csv and pip3 install csv, but neither of those worked. I get the following error: ERROR: Could not find a version that satisfies the requirement csv (from versions: none) ERROR: No matching distribution found for csv  I also tried downloading csv by going to Pycharm's setting and adding csv as a project interpreter, but it gives me the same error message. Hopefully, you can help me, thanks!

Answer: You can't install the csv module because it is already part of Python, so you can just start to use it by including in your file: import csv  .
Question: I have a data frame with a date column which is a timestamp. There are multiple data points per hour of a day eg 2014-1-1 13:10, 2014-1-1 13:20 etc. I want to group the data points from the same hour of a specific day and then create a heatmap using seaborn and plot a different column. I have tried to use groupby but I'm not sure how to specify i want the hour and day date data 2014-1-1 13:10 50 2014-1-1 13:20 51 2014-1-1 13:30 51 2014-1-1 13:40 56 2014-1-1 13:50 67 2014-1-1 14:00 43 2014-1-1 14:10 78 2014-1-1 14:20 45 2014-1-1 14:30 58  I want to combine the data by its mean value

Answer: You can use dt.strftime('%H') to get the hours, and dt.strftime('%Y-%m-%D') or dt.normalize() for the days sns.heatmap(df.groupby([df.date.dt.normalize(), df.date.dt.strftime('%H:00')]) ['data'].mean() .rename_axis(index=['day','hour']) .unstack(level=0) )  Output:   Update: for the weeks, we can use a similar approach s = (df.groupby([df.date.dt.isocalendar().week, df.date.dt.strftime('%Y-%m-%d'), df.date.dt.strftime('%H:00')]) ['data'].mean() .rename_axis(index=['week','day','hour']) ) fig, axes = plt.subplots(2,2, figsize=(10,10)) for w, ax in zip(s.index.unique('week'), axes.ravel()): sns.heatmap(s.loc[w].unstack(level='day'), ax=ax) ax.set_title(f'Week {w}')  Output:
Question: I have a point point = np.array([0.07852388, 0.60007135, 0.92925712, 0.62700219, 0.16943809, 0.34235233])  And a pandas dataframe  a b c d e f 0 0.025641 0.554686 0.988809 0.176905 0.050028 0.333333 1 0.027151 0.520914 0.985590 0.409572 0.163980 0.424242 2 0.028788 0.478810 0.970480 0.288557 0.095053 0.939394 3 0.018692 0.450573 0.985910 0.178048 0.118399 0.484848 4 0.023256 0.787253 0.865287 0.217591 0.205670 0.303030  I would like to calculate the distance of every row in the pandas dataframe, to that specific point I tried import numpy as np d_all = list() for index, row in df_scaled[cols_list].iterrows(): d = np.linalg.norm(centroid-np.array(list(row[cols_list]))) d_all += [d] df_scaled['distance_cluster'] = d_all  My solution is really slow though (taking into account that I want to calculate the distance from other points as well. Is there a way to do my calculations more efficiently ?

Answer: You can compute vectorized Euclidean distance (L2 norm) using the formula  sqrt((a1 - b1)2 + (a2 - b2)2 + ...)  df.sub(point, axis=1).pow(2).sum(axis=1).pow(.5) 0 0.474690 1 0.257080 2 0.703857 3 0.503596 4 0.461151 dtype: float64  Which gives the same output as your current code.  Or, using linalg.norm: np.linalg.norm(df.to_numpy() - point, axis=1) # array([0.47468985, 0.25707985, 0.70385676, 0.5035961 , 0.46115096])
Question: I have a pandas dataframe of the following structure: df = pd.DataFrame({'ID':['A001', 'A001', 'A001', 'A002', 'A002', 'A003', 'A003', 'A004', 'A004', 'A004', 'A005', 'A005'], 'Val1':[2, 2, 2, 5, 6, 8, 8, 3, 3, 3, 7, 7], 'Val2':[100, -100, 50, -40, 40, 60, -50, 10, -10, 10, 15, 15]})   ID Val1 Val2 0 A001 2 100 1 A001 2 -100 2 A001 2 50 3 A002 5 -40 4 A002 6 40 5 A003 8 60 6 A003 8 -50 7 A004 3 10 8 A004 3 -10 9 A004 3 10 10 A005 7 15 11 A005 7 15  I want to remove duplicate rows where ID and Val1 are duplicates, and where Val2 sums to zero across two rows. The positive/negative Val2 rows may not be consecutive either, even under a groupby In the above sample data, rows 0 and 1, as well as 7, 8, 9 fulfill these criteria. I'd want to remove [0, 1], and either [7, 8] or [8, 9]. Another constraint here is that there could be entirely duplicate rows ([10, 11]). In this case, I want to keep both rows. The desired output is thus:  ID Val1 Val2 2 A001 2 50 3 A002 5 -40 4 A002 6 40 5 A003 8 60 6 A003 8 -50 9 A004 3 10 10 A005 7 15 11 A005 7 15  Short of iterating over each row and looking for other rows which fit the criteria, I'm out of ideas for a more "pythonic" way to do this. Any help is much appreciated.

Answer: I put some comments in the code, so hopefully, my line of thought should be clear : cond = df.assign(temp=df.Val2.abs()) # a way to get the same values (differentiated by their sign) # to follow each other cond = cond.sort_values(["ID", "Val1", "temp"]) # cumsum should yield a zero for numbers that are different # only by their sign cond["check"] = cond.groupby(["ID", "temp"]).Val2.cumsum() cond["check"] = np.where(cond.check != 0, np.nan, cond.check) # the backward fill here allows us to assign an identifier # to the two values that summed to zero cond["check"] = cond["check"].bfill(limit=1) # this is where we implement your other condition # essentially, it looks for rows that are duplicates # and rows that any two rows sum to zero cond.loc[ ~(cond.duplicated(["ID", "Val1"], keep=False) & (cond.check == 0)), ["ID", "Val1", "Val2"], ] ID Val1 Val2 2 A001 2 50 3 A002 5 -40 4 A002 6 40 6 A003 8 -50 5 A003 8 60 9 A004 3 10
Question: Do you know what are the hex codes or RGB values of the "tab" palette (the default 10 colors: tab:blue, tab:orange, etc...) of matplotlib ? And possibly do you know how if there's a way to obtain the hex code for any named color in matplotlib ?

Answer: Turns out this piece of code from the matplotlib examples gave me the answer I was after. The hex codes of the "tableau" palette are as follows: tab:blue : #1f77b4 tab:orange : #ff7f0e tab:green : #2ca02c tab:red : #d62728 tab:purple : #9467bd tab:brown : #8c564b tab:pink : #e377c2 tab:gray : #7f7f7f tab:olive : #bcbd22 tab:cyan : #17becf   Using the following code I made a dictionnary containing all the named colors and their respective hex code : import matplotlib.colors as mcolors mcolors.TABLEAU_COLORS mcolors.XKCD_COLORS mcolors.CSS4_COLORS #Base colors are in RGB so they need to be converted to HEX BASE_COLORS_hex = {name:mcolors.rgb2hex(color) for name,color in mcolors.BASE_COLORS.items()} all_named_colors = {} all_named_colors.update(mcolors.TABLEAU_COLORS) all_named_colors.update(BASE_COLORS_hex) all_named_colors.update(mcolors.CSS4_COLORS) all_named_colors.update(mcolors.XKCD_COLORS) #print(all_named_colors) print(all_named_colors["tab:blue"]) >>> #1f77b4
Question: This is continue to this question. I have added a model to get query params to pydantic model class QueryParams(BaseModel): x: str = Field(description="query x") y: str = Field(description="query y") z: str = Field(description="query z") @app.get("/test-query-url/{test_id}") async def get_by_query(test_id: int, query_params: QueryParams = Depends()): print(test_id) print(query_params.dict(by_alias=True)) return True  it is working as expected but description(added in model) is not reflecting in swagger ui  But if same model is used for request body, then description is shown in swagger  Am I missing anything to get the description for QueryParams(model) in swagger ui?

Answer: This is not possible with Pydantic models The workaround to get the desired result is to have a custom dependency class (or function) rather than the Pydantic model from fastapi import Depends, FastAPI, Query app = FastAPI() class CustomQueryParams: def __init__( self, foo: str = Query(..., description="Cool Description for foo"), bar: str = Query(..., description="Cool Description for bar"), ): self.foo = foo self.bar = bar @app.get("/test-query/") async def get_by_query(params: CustomQueryParams = Depends()): return params Thus, you will have the doc as,   References  Validate GET parameters in FastAPI--(FastAPI GitHub) It seems like there is less interest in extending the Pydantic model to validate the GET parameters Classes as Dependencies--(FastAPI Doc)
Question: Let's say that I have the following dataframe: index K1 K2 D1 D2 D3 N1 0 1 12 4 6 N2 1 1 10 2 7 N3 0 0 3 5 8  Basically, I want to transform this dataframe into the following: index COL1 COL2 K1 D1 = 0*12+1*10+0*3 K1 D2 = 0*4+1*2+0*5 K1 D3 = 0*6+1*7+0*8 K2 D1 = 1*12+1*10+0*3 K2 D2 = 1*4+1*2+0*5 K2 D3 = 1*6+1*7+0*8  The content of COL2 is basically the dot product (aka the scalar product) between the vector in index and the one in COL1. For example, let's take the first line of the resulting df. Under index, we have K1 and, under COL1 we have D1. Looking at the first table, we know that K1 = [0,1,0] and D1 = [12,10,3]. The scalar product of these two "vectors" is the value inside COL2 (first line). I'm trying to find a way of doing this without using a nested loop (because the idea is to make something efficient), however, I don't exactly know how. I tried using the pd.melt() function and, although it gets me closer to what I want, it doesn't exactly get me to where I want. Could you give me a hint?

Answer: This is matrix multiplication: (df[['D1','D2','D3']].T@df[['K1','K2']]).unstack().reset_index()  Output:  level_0 level_1 0 0 K1 D1 10 1 K1 D2 2 2 K1 D3 7 3 K2 D1 22 4 K2 D2 6 5 K2 D3 13
Question: I want to reverse the packing of the following code: struct.pack("<"+"I"*elements, *self.buf[:elements])  I know "<" means little endian and "I" is unsigned int. How can I use struct.unpack to reverse the packing?

Answer: struct.pack takes non-byte values (e.g. integers, strings, etc.) and converts them to bytes. And conversely, struct.unpack takes bytes and converts them to their 'higher-order' equivalents. For example: >>> from struct import pack, unpack >>> packed = pack('hhl', 1, 2, 3) >>> packed b'\x00\x01\x00\x02\x00\x00\x00\x03' >>> unpacked = unpack('hhl', packed) >>> unpacked (1, 2, 3)  So in your instance, you have little-endian unsigned integers (elements many of them). You can unpack them using the same structure string (the '<' + 'I' * elements part) - e.g. struct.unpack('<' + 'I' * elements, value). Example from: https://docs.python.org/3/library/struct.html
Question: I am trying to make a contour plot of my 2d data. However, I would like to input the contours manually. I found the "levels" option in seaborn.kde documentation, where I can define the levels for contours manually. However, I have no idea what these levels mean. The documentation gives this definition -  Levels correspond to iso-proportions of the density.  What does iso-proportions of density mean? Are there any references that I could read up on this?

Answer: Basically, the contour line for the level corresponding to 0.05 is drawn such that 5% of the distribution lies "below" it. Alternately, because the integral over the full density equals 1 (that's what makes it a PDF), the integral over the area outside of the contour line will be 0.05.
Question: I am trying to remove words that have length below 2 and any word that is numbers. For example  s = " This is a test 1212 test2"  Output desired is " This is test test2"  I tried \w{2,} this removes all the word whose length is below 2. When I added \D+ this removes all numbers when I didn't want to get rid of 2 from test2.

Answer: You may use: s = re.sub(r'\b(?:\d+|\w)\b\s*', '', s)  RegEx Demo Pattern Details:  \b: Match word boundary (?:\d+|\w): Match a single word character or 1+ digits \b: Match word boundary \s*: Match 0 or more whitespaces
Question: I have a dataframe a pandas dataframe with the following columns: df = pd.DataFrame([ ['A2', 2], ['B1', 1], ['A1', 2], ['A2', 1], ['B1', 2], ['A1', 1]], columns=['one','two'])  Which I am hoping to sort primarily by column 'two', then by column 'one'. For the secondary sort, I would like to use a custom sorting rule that will sort column 'one' by the alphabetic character [A-Z] and then the trailing numeric number [0-100]. So, the outcome of the sort would be: one two A1 1 B1 1 A2 1 A1 2 B1 2 A2 2  I have sorted a list of strings similar to column 'one' before using a sorting rule like so: def custom_sort(value): return (value[0], int(value[1:])) my_list.sort(key=custom_sort)  If I try to apply this rule via a pandas sort, I run into a number of issues including:  The pandas DataFrame.sort_values() function accepts a key for sorting like the sort() function, but the key function should be vectorized (per the pandas documentation). If I try to apply the sorting key to only column 'one', I get the error "TypeError: cannot convert the series to <class 'int'>" When you use the pandas DataFrame.sort_values() method, it applies the sort key to all columns you pass in. This will not work since I want to sort first by the column 'two' using a native numerical sort.  How would I go about sorting the DataFrame as mentioned above?

Answer: You can split column one into its constituent parts, add them as columns to the dataframe and then sort on them with column two. Finally, remove the temporary columns. >>> (df.assign(lhs=df['one'].str[0], rhs=df['one'].str[1:].astype(int)) .sort_values(['two', 'rhs', 'lhs']) .drop(columns=['lhs', 'rhs'])) one two 5 A1 1 1 B1 1 3 A2 1 2 A1 2 4 B1 2 0 A2 2
Question: In built-in python's logging module you create a logger and log messages: import logging log = logging.getLogger('mylogger') log.info('starting!')  You can also pass extras to the log message that will be used later by the formatter: user = 'john doe' log.info('starting!', extra={'user': 'john doe'})  However it's quite tedious to do it explicitly with every message. Is it possible to set extras for the whole logger, so they would be passed with every log? Something like: log = logging.getLogger('mylogger') log.extras = {'user': 'john doe'} log.info('starting!')  Currently the only way it seems is to monkey patch log() method: def patch_default_extras(logger, extras): original_log = logger.log def log_with_extras(level, msg, *args, **kwargs): kwargs['extra'] = {**extras, **kwargs.get('extra', {})} return original_log(level, msg, *args, **kwargs) logger.log = log_with_extras mylog = logging.getLogger('mylogger') patch_default_extras(mylog, {'user': 'john doe'}) mylog = logging.getLogger('mylogger') mylog.log(logging.ERROR, 'hello')  Which is rather ugly, but unless I'm missing something ‒ it's the only way?

Answer: The builtin way to do this is using a logging adapter. A filter would also work but for your described use-case adapters are perfect. import logging logging.basicConfig(level=logging.DEBUG, format='user: %(user)s - message: %(msg)s') logger = logging.getLogger() logger_with_user = logging.LoggerAdapter(logger, {'user': 'jane doe'}) logger_with_user.info('test') # user: jane doe - message: test
Question: I need to generate as fast as possible all permutations of integers 0, 1, 2, ..., n - 1 and have result as a NumPy array of shape (factorial(n), n), or to iterate through large portions of such an array to conserve memory. Is there some built-in function in NumPy for doing this? Or some combination of functions. Using itertools.permutations(...) is too slow, I need a faster method.

Answer: Here's a NumPy solution that builds the permutations of size m by modifying the permutations of size m-1 (see more explanation further down): from math import factorial def permutations(n): a = np.zeros((factorial(n), n), np.uint8) f = 1 for m in range(2, n+1): b = a[:f, n-m+1:] # the block of permutations of range(m-1) for i in range(1, m): a[i*f:(i+1)*f, n-m] = i a[i*f:(i+1)*f, n-m+1:] = b + (b >= i) b += 1 f *= m return a  Demo: >>> permutations(3) array([[0, 1, 2], [0, 2, 1], [1, 0, 2], [1, 2, 0], [2, 0, 1], [2, 1, 0]], dtype=uint8)  For n=10, the itertools solution takes 5.5 seconds for me while this NumPy solution takes 0.2 seconds. How it proceeds: It starts with a zero-array of the goal size, which already contains the permutations for range(1) at the top-right (I "dotted out" the other parts of the array): [[. . 0] [. . .] [. . .] [. . .] [. . .] [. . .]]  Then it turns that into the permutations of range(2): [[. 0 1] [. 1 0] [. . .] [. . .] [. . .] [. . .]]  And then into the permutations of range(3): [[0 1 2] [0 2 1] [1 0 2] [1 2 0] [2 0 1] [2 1 0]]  It does so by filling the next-left column and by copying/modifying the previous block of permutations downwards.
Question: I installed psycopg2 using conda on Windows 10. https://anaconda.org/anaconda/psycopg2 I did it in a clean new conda environment (named wr). I then tried to run this sample app but I am getting this error (see below). I have no idea what I might be doing wrong because it was all straightforward and I did it in a clean way. Any ideas how to solve this? import psycopg2 try: connection = psycopg2.connect(user = "***", password = "***", host = "***", port = "5432", database = "***") cursor = connection.cursor() # Print PostgreSQL Connection properties print ( connection.get_dsn_parameters(),"\n") # Print PostgreSQL version cursor.execute("SELECT version();") record = cursor.fetchone() print("You are connected to - ", record,"\n") except (Exception, psycopg2.Error) as error : print ("Error while connecting to PostgreSQL", error) finally: #closing database connection. if(connection): cursor.close() connection.close() print("PostgreSQL connection is closed")  Error in VS code: PS C:\Work\WRR\git\tools\JTunnelTestApp> cd 'c:\Work\WRR\git\tools\JTunnelTestApp'; & 'C:\Programs\Anaconda3\envs\wr\python.exe' 'c:\Users\petrop01\.vscode\extensions\ms-python.python-2020.9.114305\pythonFiles\lib\python\debugpy\launcher' '56143' '--' 'c:\Work\WRR\git\tools\JTunnelTestApp\main.py' Traceback (most recent call last): File "c:\Work\WRR\git\tools\JTunnelTestApp\main.py", line 1, in <module> import psycopg2 File "C:\Programs\Anaconda3\envs\wr\lib\site-packages\psycopg2\__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa ImportError: DLL load failed while importing _psycopg: The operating system cannot run %1. PS C:\Work\WRR\git\tools\JTunnelTestApp>  EDIT: Seems they had a bug open for this 2 years ago and they just closed it, ignoring it completely. https://github.com/psycopg/psycopg2/issues/734

Answer: You can use psycopg2-binary library instead of psycopg2. After installation the usage is the same. psycopg2 requires some dll file to be installed on your operating system. You can either install it manually or you can install psycopg2-binary instead which packs both the library and dll files together. The documentation resides in psycopg2 library docs: https://psycopg.org/docs/install.html
Question: I've added a local directory to my system path. When I import it, it's fine but when I do import local_repo print(local_repo.__file__)  it returns None. How do I get it to return the path... P.S. When I try this with other modules works fine - returns the path - for example import pathlib print(pathlib.__file__) >>>> "C:\Python38\lib\pathlib.py" 

Answer: __file__ is None because there is no file for your package. You've made a namespace package, and those aren't even supposed to correspond to a specific directory, let alone a file - they're designed for a very specialized use case that doesn't match what you're doing. If you want a regular package, with a non-None __file__ and all, you should give it an __init__.py file. __file__ will then be the path to the package's __init__.py.
Question: how can I change tf.contrib.layers.xavier_initializer()  to tf version >= 2.0.0 ?? all codes: W1 = tf.get_variable("W1", shape=[self.input_size, h_size], initializer=tf.contrib.layers.xavier_initializer()) 

Answer: the TF2 replacement for tf.contrib.layers.xavier_initializer() is tf.keras.initializers.glorot_normal(Xavier and Glorot are 2 names for the same initializer algorithm, referring to one researcher named Xavier Glorot) documentation link. if dtype is important for some compatibility reasons - use tf.compat.v1.keras.initializers.glorot_normal
Question: I am attempting to run a SQL query against a Redshift DB in a Python script. The following results in an error "TypeError: dict is not a sequence" and I cannot figure out why. test1 = """ WITH A AS ( SELECT *, CASE WHEN substring(text_value, 0, 4) LIKE ' ' THEN substring(substring(text_value, 0, 4), 3) ELSE substring(text_value, 0, 4) END AS cost_center_id FROM job_custom_fields WHERE key = 'cost_center' ) SELECT job_id, display_value, CASE WHEN cost_center_id LIKE '%T%' THEN SUBSTRING(cost_center_id, 1,1) WHEN cost_center_id LIKE '%D%' THEN SUBSTRING(cost_center_id, 1,1) WHEN cost_center_id LIKE '%P%' THEN SUBSTRING(cost_center_id, 1,1) ELSE cost_center_id END AS cost_center_id FROM A """ red_engine = create_engine('postgresql+psycopg2://org_525:LZynsTS56PqPlHVhIIgUdDf1c1wuW4gD@redshift.greenhouse.io:5439/greenhouse') test = pd.read_sql_query(test1,red_engine) test  Any assistance is much appreciated

Answer: you need to escape % character using %% in Python. % is used for string formatting in Python. Not a solution to this problem but it's worth mentioning here that in Python 3.6 and later, there is a more recommended way of string formatting using f-strings. For example: name = 'eshirvana' message = f"I am {name}" print(message)