eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: We know that MongoDB has two ways of modeling relationships between relations/entities, namely, embedding and referencing (see difference here). Let's say we have a USER database with two tables in mySQL named user and address. An embedded MongoDB document might look like this: { "_id": 1, "name": "Ashley Peacock", "addresses": [ { "address_line_1": "10 Downing Street", "address_line_2": "Westminster", "city": "London", "postal_code": "SW1A 2AA" }, { "address_line_1": "221B Baker Street", "address_line_2": "Marylebone", "city": "London", "postal_code": "NW1 6XE" } ] }  Whereas in a referenced relation, 2 SQL tables will make 2 collections in MongoDB which can be migrated by this apporoach using pymongo. How can we directly migrate MySQL data as an embedded document using python? Insights about about Pseudo code and performance of algorithm will be highly useful. Something that comes to my mind is creating views by performing joins in MySQL. But in that case we will not be having the structure of children document inside a parent document.

Answer: Denormalization First, for canonical reference, the question of "embedded" vs. "referenced" data is called denormalization. Mongo has a guide describing when you should denormalize. Knowing when and how to denormalize is a very common hang-up when moving from SQL to NoSQL and getting it wrong can erase any performance benefits you might be looking for. I'll assume you've got this figured out since you seem set on using an embedded approach. MySQL to Mongo Mongo has a great Python tutorial you may want to refer to. First join your user and address tables. It will look something like this: | _id | name | address_line_1 | address_line_2 | ... | 1 | Ashley Peacock | 10 Downing Street ... | ... | 1 | Ashley Peacock | 221B Baker Street ... | ... | 2 | Bob Jensen | 343 Main Street ... | ... | 2 | Bob Jensen | 1223 Some Ave ... | ... ...  Then iterate over the rows to create your documents and pass them to pymongo insert_one. Using upsert=True with insert_one will insert a new document if a matching one is not found in the database, or update an existing document if it is found. Using $push appends the address data to the array field addresses in the document. With this setup, insert_one will automatically handle duplicates and append addresses based on matching _id fields. See the docs for more details: from pymongo import MongoClient client = MongoClient(port=27017) db = client.my_db sql_data = [] # should have your SQL table data # depending on how you got this into python, you will index with a # field name or a number, e.g. row["id"] or row[0] for row in sql_data: address = { "address_line_1": row["address_line_1"], "address_line_2": row["address_line_2"], "city": row["city"], "postal_code": row["postal_code"], } db.users.update_one( {"_id": row["_id"]}, {"name": row["name"], "$push": {"addresses": address}}, upsert=True, )
Question: I am trying to create a google credentials object to access google drive. I have the tokens and user data stored in a session thus I am trying to create the object without loading them from a credentials.json file. I am handling the authentication when a user first logs in the web app and storing the tokens inside the user session, the session has a time out of 24 hours same as the default access token thus after 24 hours the user is requested to log in again so a new session with a valid access token is created. So my idea is to reuse the access token to limit the amount of log ins and improve user experience. This is a small piece of code on how I'm trying to create the google credentials object from oauth2client.client import GoogleCredentials access_token = request.session['access_token'] gCreds = GoogleCredentials( access_token, os.getenv('GOOGLE_CLIENT_ID'), os.getenv('GOOGLE_CLIENT_SECRET'), refresh_token=None, token_expiry=None, token_uri=GOOGLE_TOKEN_URI, user_agent='Python client library', revoke_uri=None) build('drive', 'v3', credentials = gCred)  Whenever I try to run this code I get the following error:  Insufficient Permission: Request had insufficient authentication scopes.". Details: "[{'domain': 'global', 'reason': 'insufficientPermissions', 'message': 'Insufficient Permission: Request had insufficient authentication scopes.'}]" { "error": { "errors": [ { "domain": "usageLimits", "reason": "dailyLimitExceededUnreg", "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.", "extendedHelp": "https://code.google.com/apis/console" } ], "code": 403, "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup." } } 

Answer: The object was build correctly, there was no issue with building the object this way access_token = request.session['access_token'] gCreds = GoogleCredentials( access_token, os.getenv('GOOGLE_CLIENT_ID'), os.getenv('GOOGLE_CLIENT_SECRET'), refresh_token=None, token_expiry=None, token_uri=GOOGLE_TOKEN_URI, user_agent='Python client library', revoke_uri=None) build('drive', 'v3', credentials = gCred)  The problem was caused at an earlier stage, when registering the scopes oauth = OAuth(config) oauth.register( name='google', server_metadata_url='', client_kwargs={ 'scope': 'openid email profile https://www.googleapis.com/auth/drive.readonly' } )
Question: I've implemented Stripe checkout on a Django app and it's all working correctly except that it's not showing up on the Stripe Dashboard, even though it's showing in the event data on the same page. Have I formatted it incorrectly or am I overlooking something obvious? This is how I added meta data:  checkout_session = stripe.checkout.Session.create( payment_method_types=['card'], line_items = line_itemz, metadata={ "payment_type":"schedule_visit", "visit_id":visit_id }, mode='payment', success_url= 'http://localhost:8000/success', cancel_url = 'http://localhost:8000/cancel',)  Here is a screenshot of the Metadata section empty, but in the events the Metadata is there as it should be:  Again I can access the metadata every where else but would like it to show up on the dashboard so my team can more easily access that information.

Answer: The metadata field you set is for Checkout Session alone, but not on Payment Intent (which is the Dashboard page you are at). To have metadata shown at the Payment Intent, I'd suggest also setting payment_intent_data.metadata [0] in the request when creating a Checkout Session. For example, session = stripe.checkout.Session.create( success_url="https://example.com/success", cancel_url="https://example.com/cancel", line_items=[ { "price": "price_xxx", "quantity": 1, }, ], mode="payment", metadata={ "payment_type": "schedule_visit", "visit_id": "123" }, payment_intent_data={ "metadata": { "payment_type": "schedule_visit", "visit_id": "123" } } )  [0] https://stripe.com/docs/api/checkout/sessions/create#create_checkout_session-payment_intent_data-metadata
Question: I have a DataFrame where the index is a DatetimeIndex with a daily frequency. It contains 365 rows, one for each day of the year. When computing rolling sums, the first few elements are always NaN (as expected), but I'd like them to have actual values. For example, if using a rolling window of 3 samples, the value for Jan 1 should be the sum of Dec 30, Dec 31, and Jan 1. Similarly, the value for Jan 2 should be the sum of Dec 31, Jan 1, and Jan 2. I have looked into all parameters of the rolling function in Pandas, and could not find anything that would provide this wrapping. Any help would be appreciated. The code below is a minimal example illustrating the behavior of rolling. import numpy as np import pandas as pd fake_data = pd.DataFrame(index=pd.date_range('2022-1-1', '2022-12-31', freq='D'), data=np.random.random(365)) rolling_fake_data = fake_data.rolling(3).sum() 

Answer: You're basically asking for a circular data object which has no start or end. Not sure that exists! The best work-around I can think of is to repeat the end of the series before the beginning. n = 3 rolling_fake_data = ( pd.concat([fake_data[-n:], fake_data]) ).rolling(n).sum()[n:] # Test assert(rolling_fake_data.loc["2022-01-01", 0] == fake_data.loc[["2022-12-30", "2022-12-31", "2022-01-01"], 0].sum())
Question: I have the following input file in csv A,B,C,D 1,2,|3|4|5|6|7|8,9 11,12,|13|14|15|16|17|18,19  How do I split column C right in the middle into two new rows with additional column E where the first half of the split get "0" in Column E and the second half get "1" in Column E? A,B,C,D,E 1,2,|3|4|5,9,0 1,2,|6|7|8,9,1 11,12,|13|14|15,19,0 11,12,|16|17|18,19,1  Thank you

Answer: Here's how to do it without Pandas: import csv with open("input.csv", newline="") as f_in, open("output.csv", "w", newline="") as f_out: reader = csv.reader(f_in) header = next(reader) # read header header += ["E"] # modify header writer = csv.writer(f_out) writer.writerow(header) for row in reader: a, b, c, d = row # assign 4 items for each row c_items = [x.strip() for x in c.split("|") if x.strip()] n_2 = len(c_items) // 2 # halfway index c1 = "|" + "|".join(c_items[:n_2]) c2 = "|" + "|".join(c_items[n_2:]) writer.writerow([a, b, c1, d, 0]) # 0 & 1 will be converted to str on write writer.writerow([a, b, c2, d, 1])
Question: I have a function that converts a value in one format to another. It's analogous to converting Fahrenheit to Celsius for example. Quite simply, the formula is: l = -log(20/x)  I am inheriting SAS code from a colleague that has the following hardcoded for a range of values of x: "if x= 'a' then x=l;"  which is obviously tedious and limited in scope. How best could I convert this to a function that could be called in a SAS script? I previously had it in Python as: def function(x): l = -np.log10(20/float(x)) return l  and then would simply call the function. Thank you for your help - I'm adusting from Python to SAS and trying to figure out how to make the switch.

Answer: If you are interested in writing your own functions, as Joe said, proc fcmp is one way to do it. This will let you create functions that behave like SAS functions. It's about as analogous to Python functions as you'll get. It takes a small bit of setup, but it's really nice in that the functions are all saved in a SAS dataset that can be transferred from environment to environment. The below code creates a function called f() that does the same as the Python function. proc fcmp outlib=work.funcs.log; function f(x); l = log10(20/x); return(l); endfunc; run; options cmplib=work.funcs;  This is doing three things:  Creating a function called f() that takes one input, x Saving the function in a dataset called work.funcs that holds all functions Labeling all the functions under the package log  Don't worry too much about the label. It's handy if you have many different function packages you want, for example: time, dates, strings, etc. It's helpful for organization, but it is a required label. Most of the time I just do work.funcs.funcs. options cmplib=work.funcs says to load the dataset funcs which holds all of your functions of interest. You can test your function below: data test; l1 = f(1); l2 = f(2); l10 = f(10); run;  Output: l1 l2 l10 1.3010299957 1 0.3010299957  Also, SAS does have a Python interface. If you're more comfortable programming in Python, take a look at SASPy to get all the benefits of both SAS and Python.
Question: I have a 3D array consist of 2D arrays, I want to rotate only the 2D arrays inside the 3D array without changing the order, so it will become a 3D array consist of rotated 3D arrays. For example, I have a 3D array like this. foo = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]) print(foo) >>> array([[[ 1, 2, 3], [ 4, 5, 6]], [[ 7, 8, 9], [10, 11, 12]]]) foo.shape >>> (2, 2, 3)  I want to rotate it into this. rotated_foo = np.array([[[4, 1], [5, 2], [6, 3]], [[10, 7], [11, 8], [12, 9]]]) print(rotated_foo) >>> array([[[ 4, 1], [ 5, 2], [ 6, 3]], [[10, 7], [11, 8], [12, 9]]]) rotated_foo.shape >>> (2, 3, 2)  I've tried it using numpy's rot90 but I got something like this. rotated_foo = np.rot90(foo) print(rotated_foo) >>> array([[[ 4, 5, 6], [10, 11, 12]], [[ 1, 2, 3], [ 7, 8, 9]]]) rotated_foo.shape >>> (2, 2, 3) 

Answer: You can use numpy.rot90 by setting axes that you want to rotate. foo = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]) rotated_foo = np.rot90(foo, axes=(2,1)) print(rotated_foo)  Output: array([[[ 4, 1], [ 5, 2], [ 6, 3]], [[10, 7], [11, 8], [12, 9]]])
Question: I am aware that using shutil.copy(src,dst) copies files and shutil.copytree(src,dst) is used for copying directories. Is there any way so I don't have to differentiate between copying folders and copying files? Tysm

Answer: You might want to take a look to this topic, where the same question was answered. https://stackoverflow.com/a/1994840/17595642 Functions can be written to do so. Here is the one implemented in the other topic : import shutil, errno def copyanything(src, dst): try: shutil.copytree(src, dst) except OSError as exc: # python >2.5 if exc.errno in (errno.ENOTDIR, errno.EINVAL): shutil.copy(src, dst) else: raise
Question: I have my pandas dataframe and i need to find the index of a certain value. But the thing is, this df does from an other one where i had to cut some part using the df.loc, so it ends up like : index value 1448 31776 1449 32088 1450 32400 1451 32712 1452 33024  Let's say i need to find the index of the value '32400', with the function df.index[df.value == 32400] i get 1450, but what I want is 2. Would you know how i could get this value?

Answer: First idea is create default index starting by 0 by DataFrame.reset_index with drop=True: df = df.reset_index(drop=True) idx = df.index[df.value == 32400] print (idx) Int64Index([2], dtype='int64')  Or use numpy - e.g. by numpy.where with condition for positions of match: idx = np.where(df.value == 32400)[0] print (idx) [2]  Or use numpy.argmax - it working well if match at least one value: idx = np.argmax(df.value == 32400) print (idx) 2 idx = np.argmax(df.value == 10) print (idx) 0
Question: I have a dataframe (df) with the following structure:     retweet_datetime tweet_id tweet_datetime     2020-04-24 03:33:15 85053699 2020-04-24 02:28:22   2020-04-24 02:43:35 85053699 2020-04-24 02:28:22   2020-04-18 04:24:03 86095361 2020-04-18 00:06:01   2020-04-18 00:19:08 86095361 2020-04-18 00:06:01   2020-04-18 00:18:38 86095361 2020-04-18 00:06:01   2020-04-18 00:07:08 86095361 2020-04-18 00:06:01     The retweet_datetime is sorted from latest to newest retweets. I'd like to create two new columns as follows:  tweet_lifetime1: the difference between the last retweet time and the first retweet time, i.e., for each tweet_id: last retweet_datetime - first retweet_datetime tweet_lifetime2: the difference between the last retweet time and tweet creation time (tweet_datetime)  Update For example, for the tweet id: "86095361":  tweet_lifetime1 = 2020-04-18 04:24:03 - 2020-04-18 00:07:08 (04:16:55) tweet_lifetime2 = 2020-04-18 04:24:03 - 2020-04-18 00:06:01 (04:18:02)  The expected output df:     retweet_datetime tweet_id tweet_datetime lifetime1 lifetime2     2020-04-24 03:33:15 85053699 2020-04-24 02:28:22 00:49:40 01:04:53   2020-04-18 04:24:03 86095361 2020-04-18 00:06:01 04:16:55 04:18:02     I've seen several similar posts, but they mostly subtract consecutive rows. For example, I can subtract the time difference between each retweet_datetimes for each tweet id as follows: df2 = df.assign(delta = df.groupby('tweet_id')['retweet_datetime'].diff()) 

Answer: Use named aggregation with subtract column with Series.sub, DataFrame.pop is used for drop column tmp after processing: df1 = (df.groupby('tweet_id', as_index=False) .agg(retweet_datetime=('retweet_datetime','first'), tmp = ('retweet_datetime','last'), tweet_datetime = ('tweet_datetime','last'))) df1['lifetime1'] = df1['retweet_datetime'].sub(df1.pop('tmp')) df1['lifetime2'] = df1['retweet_datetime'].sub(df1['tweet_datetime']) print (df1) tweet_id retweet_datetime tweet_datetime lifetime1 \ 0 85053699 2020-04-24 03:33:15 2020-04-24 02:28:22 0 days 00:49:40 1 86095361 2020-04-18 04:24:03 2020-04-18 00:06:01 0 days 04:16:55 lifetime2 0 0 days 01:04:53 1 0 days 04:18:02  If need format HH:MM:SS use: def f(x): ts = x.total_seconds() hours, remainder = divmod(ts, 3600) minutes, seconds = divmod(remainder, 60) return ('{:02d}:{:02d}:{:02d}').format(int(hours), int(minutes), int(seconds)) df1['lifetime1'] = df1['retweet_datetime'].sub(df1.pop('tmp')).apply(f) df1['lifetime2'] = df1['retweet_datetime'].sub(df1['tweet_datetime']).apply(f) print (df1) tweet_id retweet_datetime tweet_datetime lifetime1 lifetime2 0 85053699 2020-04-24 03:33:15 2020-04-24 02:28:22 00:49:40 01:04:53 1 86095361 2020-04-18 04:24:03 2020-04-18 00:06:01 04:16:55 04:18:02
Question: I need to find the pairs (i,j) and number of pairs for a number N such that the below conditions are satisfied: 1 <= i <= j <= N and also i * i * i = j * j. For example, for N = 50, number of pairs is 3 i.e., (1,1), (4,8), (9,27). I tried the below function code but it takes too much time for a large number like N = 10000 or more: def compute_pairs(N): pair = [] for i in range (1, N): for j in range (i, N): print( 'j =', j) if i*i*i == j*j: new_pair = (i,j) pair.append(new_pair) print(pair) return len(pair) 

Answer: Let k be the square root of some integer i satisfying i*i*i == j*j, where j is also an integer. Since k is the square root of an integer, k*k is integer. From the equation, we can solve that j is equal to k*k*k, so that is also an integer. Since k*k*k is an integer and k*k is an integer, it follows by dividing these two that k is rational. But k is the square root of an integer, so it must be either an integer or irrational. Since it is rational, it must be an integer. Since k is an integer, all the solutions are simply (k*k, k*k*k) for integer k. Since we will iterate over k>=1, we know that k*k <= k*k*k, i.e. i <= j, so we don't have to worry about that. We just have to stop iterating when k*k*k reaches N. from itertools import count # unbounded range; we will `break` when appropriate def compute_pairs(N): result = [] for k in count(1): i, j = k*k, k*k*k if j >= N: break result.append((i, j)) return result  This runs almost instantaneously even for N of 100000, no C-level optimization needed.
Question: I'm trying to build a Docker container that runs Python code on AWS Lambda. The build works fine, but when I test my code, I get the following error: {"errorMessage": "Unable to import module 'function': No module named 'utils'", "errorType": "Runtime.ImportModuleError", "stackTrace": []}  I basically have two python scripts in my folder, function.py and utils.py, and I import some functions from utils.py and use them in function.py. Locally, it all works fine, but when I build the Container and test it with the following curl command, I get the above error. Test curl command: curl --request POST \ --url http://localhost:9000/2015-03-31/functions/function/invocations \ --header 'Content-Type: application/json' \ --data '{"Input": 4}'  Here's my dockerfile: FROM public.ecr.aws/lambda/python:3.7 WORKDIR / COPY . . COPY function.py ${LAMBDA_TASK_ROOT} COPY requirements.txt . RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}" CMD [ "function.lambda_handler" ]  What I read in related Stackoverflow questions is to try importing the functions from utils.py in another way, I've tried changing from utils import * to from .utils import all, I changed the WORKDIR in my Dockerfile, and I put the utils file in a separate utils folder and tried importing this way: from utils.utils import *. I have also tried running the code in Python 3.8. Here's my folder structure: Here's my folder structure Does anyone know what I'm doing wrong?

Answer: The Dockerfile statement COPY . . copies all files to the working directory, which given your previous WORKDIR, is /. To resolve the Python import issue, you need to move the Python module to the right directory: COPY utils.py ${LAMBDA_TASK_ROOT}
Question: I am facing an error like No such file or directory.... Could anyone help me with this out? I'd really appreciate it! This is the directory:  And this is the setup.py code: from setuptools import setup setup( name='raw-microsoft-wwi', version='1.0.0', package_dir={"":"src"}, packages=[ "raw_microsoft_wwi", "raw_microsoft_wwi.configuration", "raw_microsoft_wwi.reader", "raw_microsoft_wwi.writer", "raw_microsoft_wwi.transformer" ], package_data = {"src/raw_microsoft_wwi/queries": ["*.sql"]}, include_package_data = True, install_requires=[ "delta-spark==1.2.0", "pyspark==3.2.1", "python-environ==0.4.54" ], zip_safe=False ) 

Answer: Setuptools expects package_data to be a dictionary mapping from Python package names to a list of file name patterns. This means that the keys in the dictionary should be similar to the values you listed in the packages configuration (instead of paths). I think you can try the following:  Add the missing raw_microsoft_wwi.queries, raw_microsoft_wwi.queries.incremental_table_queries, raw_microsoft_wwi.queries.regular_tables_queries (etc) to the packages list (effectivelly all these folders are considered packages by Python, even if they don't have a __init__.py file). Replace package_data, with something similar to:  package_data = {"raw_microsoft_wwi.queries": ["**/*.sql"]}  More information is available in https://setuptools.pypa.io/en/latest/userguide/datafiles.html.
Question: When I go to run a python code directly through the terminal it gives me this error, I've already tried to reinstall numpy and it didn't work! And I tried to install mlk service returns the same error. Can someone help me ? UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service from . import _distributor_init Traceback (most recent call last): File "c:\Users\teste.user\Desktop\Project-python\teste.py", line 4, in <module> import pandas as pd File "C:\Users\teste.user\Anaconda3\lib\site-packages\pandas\__init__.py", line 16, in <module> raise ImportError( ImportError: Unable to import required dependencies: numpy: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: * The Python version is: Python3.9 from "C:\Users\teste.user\Anaconda3\python.exe" * The NumPy version is: "1.21.5" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found. 

Answer: Can be solved by resetting package configuration by force reinstall of numpy. conda install numpy --force-reinstall
Question: How do I detect Mouse scroll up and down in python using pygame? I have created a way to detect it, but it doesn't give me any information about which way I scrolled the mouse as well as being terrible at detecting mouse scrolls where only 1 in 20 are getting detected. for event in pygame.event.get(): if event.type == pygame.MOUSEWHEEL: print("Mouse Scroll Detected.")  Any other ways I can detect mouse scrolls?

Answer: The MOUSEWHEEL event object has x and y components (see pygame.event module). These components indicate the direction in which the mouse wheel was rotated (for horizontal and vertical wheel): for event in pygame.event.get(): if event.type == pygame.MOUSEWHEEL: print(event.x, event.y)