eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: For example, I have two arrays: import numpy as np x = np.array([1,2,3]) y = np.array([10, 11])  How can I generate a pandas dataframe with every combination of x and y, like below? x y 1 10 1 11 2 10 2 11 3 10 3 11 

Answer: import pandas as pd import numpy as np x = np.array([1,2,3]) y = np.array([10, 11]) pd.DataFrame({'x':np.repeat(x,y.shape[0]), 'y':np.tile(y,x.shape[0])})  yields:  x y 0 1 10 1 1 11 2 2 10 3 2 11 4 3 10 5 3 11
Question: I am trying to pivot a table in SQL Server (52M+ observations) however I am not getting the results I need. There are 15 descriptions each with a value that I need to pivot. Original Dataframe: ID | Date | Description| Value ------------------------------------------------- P1 | 2016-12-31 | ABC | 900 P2 | 2016-11-30 | XYZ | 800 P3 | 2016-10-31 | MNO | 700  Desired Results ID | Date | ABC | XYZ | MNO ------------------------------------------------- P1 | 2016-12-31 | 900 | | P2 | 2016-11-30 | | 800 | P3 | 2016-10-31 | | | 700  I have tried pivoting this in PySpark and SQL, but have not gotten a working result. SQL Attempt: SELECT [Date] ,[ID] ,[Description] ,[Value] FROM [DB].[TABLE] WHERE ( ([Description] IN ('ABC','XYZ', 'MNO')) PIVOT( COUNT([Value]) FOR Description IN ( [ABC], [XYZ], [MNO]) ) AS pivot_table;  I tried this in Pyspark however it doesnt work either: df.groupBy("ID","Date").pivot("Description").sum("Value") 

Answer: Use conditional aggregation: select id, date, max(case when description = 'ABC' then value end) as abc, max(case when description = 'DEF' then value end) as def, max(case when description = 'MNO' then value end) as mno from mytable group by id, date
Question: Given a sha256 hash of a str in python: import hashlib hash = hashlib.sha256('foobar'.encode('utf-8'))  How can the hash be converted to a UUID? Note: there will obviously be a many-to-one mapping of hexdigest to UUID given that a hexdigest has 2^256 possible values and a UUID has 2^128. Thank you in advance for your consideration and response.

Answer: Given that UUID takes a 32 hex character input string and hexdigest produces 64 characters, a simple approach would be to sub-index the resulting hash digest to achieve the appropriate string length: import hashlib import uuid hash = hashlib.sha256('foobar'.encode('utf-8')) uuid.UUID(hash.hexdigest()[::2])
Question: I am using VS-Code and anaconda environment for python interpreter. I select the exact anaconda base environment by ctrl + shift + ` and it also reflects in the downside panel of vscode. But, when I checked the python version it shows my system's default python environment 3.7.9. If you see the below screenshot than, the anaconda environment is with 3.8.3. Please give me solution, Thank you.

Answer: To check & change vs code interpreter:  In top left menu bar Click view In the dropdown menu, Click Command Palette Click Python: Select Interpreter Choose & Click on your desired Interpreter  Another way to be sure to use anconda interpreter, open anaconda navigator and launch vs code from there. original vs code How-To
Question: Previously I used the marshmallow library with the Flask. Some time ago I have tried FastAPI with Pydantic. At first glance pydantic seems similar to masrhmallow but on closer inspection they differ. And for me the main difference between them is post_load methods which are from marshmallow. I can't find any analogs for it in pydantic. post_load is decorator for post-processing methods. Using it I can handle return object on my own, can do whatever I want: class ProductSchema(Schema): alias = fields.Str() category = fields.Str() brand = fields.Str() @post_load def check_alias(self, params, **kwargs): """One of the fields must be filled""" if not any([params.get('alias'), params.get('category'), params.get('brand')]): raise ValidationError('No alias provided', field='alias') return params  Besides it used not only for validation. Code example is just for visual understanding, do not analyze it, I have just invented it. So my question is: is there any analog for post_load in pydantic?

Answer: It is not obvious but pydantic's validator returns value of the field. Pydantic v1 There are two ways to handle post_load conversions: validator and root_validator. validator gets the field value as argument and returns its value. root_validator is the same but manipulates with the whole object. from pydantic import validator, root_validator class PaymentStatusSchema(BaseModel): order_id: str = Param(..., title="Order id in the shop") order_number: str = Param(..., title="Order number in chronological order") status: int = Param(..., title="Payment status") @validator("status") def convert_status(cls, status): return "active" if status == 1 else "inactive" @root_validator def validate_order_id(cls, values): """Check order id""" if not values.get('orderNumber') and not values.get('mdOrder'): raise HTTPException(status_code=400, detail='No order data provided') return values  By default pydantic runs validators as post-processing methods. For pre-processing you should use validators with pre argument:  @root_validator(pre=True) def validate_order_id(cls, values): """Check order id""" # some code here return values  Pydantic v2 In Pydantic v2 validators migrated: validator to field_validator and root_validator to model_validator. @field_validator('status') @classmethod def convert_status(cls, status: int): return "active" if status == 1 else "inactive" @model_validator(mode='after') @classmethod def validate_order_id(cls, values): """Check order id""" if not values.get('orderNumber') and not values.get('mdOrder'): raise HTTPException(status_code=400, detail='No order data provided') return values  For post or pre-processing use mode parameter.
Question: I've built this Telegram Bot in Python, with python-telegram-bot. It's not so complex, but I want to do some regression tests to check if everything works fine after a new feature or a change, and more generally to test specific features to find bugs/edge cases. How can I achieve this? For now, I'm doing this manually, clicking bot's keyboard and typing some text.

Answer: Have you looked to unit tests that are present in python-telegram-bot library? I think it is a good place to start. For example, in this file (historical version) you can see how to test a dialog with bot that uses ConversationHandler.
Question: I'm trying to connect to https://apis.digital.gob.cl/fl/feriados/2020, but I get an requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) error on a script that works perfectly with other URLs. The code: import requests response = requests.get('https://apis.digital.gob.cl/fl/feriados/2020') print(response.status_code) 

Answer: The issue is that the website filters out requests without a proper User-Agent, so just use a random one from MDN: requests.get("https://apis.digital.gob.cl/fl/feriados/2020", headers={ "User-Agent" : "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" })
Question: I am have imported a csv file onto my Jupyter notebook and trying to obtain all the columns names and datatypes using the info() function. However, I get the following image. Any idea how to resolve it? I can't view all the columns and datatypes, only this vague information  Thanks!

Answer: use verbose as argument to info, it gives option to print the full summary. see full documentation here You can also use show_counts (or null_counts for older pandas version since it was deprecated since pandas 1.2.0) argument to see null count information For pandas >= 1.2.0: df.info(verbose=True, show_counts=True) For pandas <1.2.0: df.info(verbose=True, null_counts=True)
Question: Suppose I have following views, from fastapi import FastAPI app = FastAPI() @app.get('/hello/') def hello_world(): return {"msg": "Hello World"} @app.get('/hello/{number}/') def hello_world_number(number: int): return {"msg": "Hello World Number", "number": number}  I have been using these functions in Flask and Django  Flask: url_for(...) Django: reverse(...)  So, how can I obtain/build the URLs of hello_world and hello_world_number in a similar way?

Answer: We have got Router.url_path_for(...) method which is located inside the starlette package Method-1: Using FastAPI instance This method is useful when you are able to access the FastAPI instance in your current context. (Thanks to @Yagizcan Degirmenci) from fastapi import FastAPI app = FastAPI() @app.get('/hello/') def hello_world(): return {"msg": "Hello World"} @app.get('/hello/{number}/') def hello_world_number(number: int): return {"msg": "Hello World Number", "number": number} print(app.url_path_for('hello_world')) print(app.url_path_for('hello_world_number', number=1)) print(app.url_path_for('hello_world_number', number=2)) # Results "/hello/" "/hello/1/" "/hello/2/"  Drawback  If we are using APIRouter, router.url_path_for('hello_world') may not work since router isn't an instance of FastAPI class. That is, we must have the FastAPI instance to resolve the URL  Method-2: Request instance This method is useful when you are able to access the Request instance (the incoming request), usually, within a view. from fastapi import FastAPI, Request app = FastAPI() @app.get('/hello/') def hello_world(): return {"msg": "Hello World"} @app.get('/hello/{number}/') def hello_world_number(number: int): return {"msg": "Hello World Number", "number": number} @app.get('/') def named_url_reveres(request: Request): return { "URL for 'hello_world'": request.url_for("hello_world"), "URL for 'hello_world_number' with number '1'": request.url_for("hello_world_number", number=1), "URL for 'hello_world_number' with number '2''": request.url_for("hello_world_number", number=2}) } # Result Response { "URL for 'hello_world'": "http://0.0.0.0:6022/hello/", "URL for 'hello_world_number' with number '1'": "http://0.0.0.0:6022/hello/1/", "URL for 'hello_world_number' with number '2''": "http://0.0.0.0:6022/hello/2/" }  Drawback  We must include the request parameter in every (or required) view to resolve the URL, which might raise an ugly feel to developers.
Question: To provide a basic example, say I wanted to write: name = str(input()) age = int(input()) print('Hi, {name}, you are {age}.')  In javascript, this would look like: console.log(`Hi, ${name}, you are ${age}.`)  I assume there is no direct implementation of template literals in Python, as I haven't found any mention on Google / DDG. If I am correct in thinking that there isn't a direct implementation, have any of you found workarounds? Thanks in advance.

Answer: You can go with formatted string literals ("f-strings") since Python 3.6 f"Hi {name}, you are {age}"  Or string formatting "Hi {}, you are {}".format(name, age) "Hi {name}, you are {age}".format(name=name, age=age)  Or format specifiers "Hi %s, you are %d" % (name, age)
Question: Currently using Ubuntu 20.04 LTS with python3.8.5. Its my first time using ubuntu with absolutely no previous knowledge of terminal.SO,would love to have a detailed answer if possible. Below is terminal output when i try importing tkinter in python3. >>> import tkinter Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.8/tkinter/__init__.py", line 36, in <module> import _tkinter # If this fails your Python may not be configured for Tk ModuleNotFoundError: No module named '_tkinter' >>>  I have reinstalled python3 and tkinter using sudo apt.But still it shows same error. When i run the same command in python IDLE it works without any error. I hope this explains my problem clearly, if any other info. is required pls reply. I also tried running the command >>>from tkinter import *

Answer: Resolved the issue it occurred because the Tkinter was installed for version 3.5 and not for the 3.8 version. For that, I installed the 3.5 version and kept only one version i.e. 3.8, and installed Tkinter again, and it worked! This is just a workaround to make things work, but the more preferred way is to create a venv and then install the particular versions of python and libraries that are needed.
Question: According to the docs, the Reduction parameter takes on 3 values - SUM_OVER_BATCH_SIZE, SUM and NONE. y_true = [[0., 2.], [0., 0.]] y_pred = [[3., 1.], [2., 5.]] mae = tf.keras.losses.MeanAbsoluteError(reduction=tf.keras.losses.Reduction.SUM) mae(y_true, y_pred).numpy() > 5.5 mae = tf.keras.losses.MeanAbsoluteError() mae(y_true, y_pred).numpy() > 2.75  What I could infer about the calculation after various trials, is this:-  when REDUCTION = SUM, Loss = Sum over all samples {(Sum of differences between y_pred and y_target vector of each sample / No of element in y_target of the sample )} = { (abs(3-0) + abs(1-2))/2 } + { (abs(2-0) + abs(5-0))/2 } = {4/2} + {7/2} = 5.5.  when REDUCTION = SUM_OVER_BATCH_SIZE, Loss = [Sum over all samples {(Sum of differences between y_pred and y_target vector of each sample / No of element in y_target of the sample )}] / Batch_size or No of Samples = [ { (abs(3-0)} + abs(1-2))/2 } + { (abs(2-0) + abs(5-0))/2 } ]/2 = [ {4/2} + {7/2} ]/2 = [5.5]/2 = 2.75.   As a result, SUM_OVER_BATCH_SIZE is nothing but SUM/batch_size. Then, why is it called SUM_OVER_BATCH_SIZE when SUM actually adds up the losses over the entire batch, while SUM_OVER_BATCH_SIZE calculates the average loss of the batch. Is my assumption regarding the workings of SUM_OVER_BATCH_SIZE and SUM at all correct?

Answer: Your assumption is correct as far as I understand. If you check the github [keras/losses_utils.py][1] lines 260-269 you will see that it does performs as expected. SUM will sum up the losses in the batch dimension, and SUM_OVER_BATCH_SIZE would divide SUM by the number of total losses (batch size). def reduce_weighted_loss(weighted_losses, reduction=ReductionV2.SUM_OVER_BATCH_SIZE): if reduction == ReductionV2.NONE: loss = weighted_losses else: loss = tf.reduce_sum(weighted_losses) if reduction == ReductionV2.SUM_OVER_BATCH_SIZE: loss = _safe_mean(loss, _num_elements(weighted_losses)) return loss  You can do a easy checking with your previous example just by adding one pair of outputs with 0 loss. y_true = [[0., 2.], [0., 0.],[1.,1.]] y_pred = [[3., 1.], [2., 5.],[1.,1.]] mae = tf.keras.losses.MeanAbsoluteError(reduction=tf.keras.losses.Reduction.SUM) mae(y_true, y_pred).numpy() > 5.5 mae = tf.keras.losses.MeanAbsoluteError() mae(y_true, y_pred).numpy() > 1.8333  So, your assumption is correct. [1]: https://github.com/keras-team/keras/blob/v2.7.0/keras/utils/losses_utils.py#L25-L84
Question: I have two numpy arrays a, b of the same shape, b has a few zeros. I would like to set an output array to a / b where b is not zero, and a otherwise. The following works, but yields a warning because a / b is computed everywhere first. import numpy a = numpy.random.rand(4, 5) b = numpy.random.rand(4, 5) b[b < 0.3] = 0.0 A = numpy.where(b > 0.0, a / b, a)  /tmp/l.py:7: RuntimeWarning: divide by zero encountered in true_divide A = numpy.where(b > 0.0, a / b, a)  Filtering the division with a mask doesn't perserve the shape, so this doesn't work: import numpy a = numpy.random.rand(4, 5) b = numpy.random.rand(4, 5) b[b < 0.3] = 0.0 mask = b > 0.0 A = numpy.where(mask, a[mask] / b[mask], a)  ValueError: operands could not be broadcast together with shapes (4,5) (14,) (4,5)  Any hints on how to avoid the warning?

Answer: Simply initialize output array with the fallback values (condition-not-satisfying values) or array and then mask to select the condition-satisfying values to assign - out = a.copy() out[mask] /= b[mask]  If you are looking for performance, we can use a modified b for the division - out = a / np.where(mask, b, 1)  Going further, super-charge it with numexpr for this specific case of positive values in b (>=0) - import numexpr as ne out = ne.evaluate('a / (1 - mask + b)')  Benchmarking  Code to reproduce the plot: import perfplot import numpy import numexpr numpy.random.seed(0) def setup(n): a = numpy.random.rand(n) b = numpy.random.rand(n) b[b < 0.3] = 0.0 mask = b > 0 return a, b, mask def copy_slash(data): a, b, mask = data out = a.copy() out[mask] /= b[mask] return out def copy_divide(data): a, b, mask = data out = a.copy() return numpy.divide(a, b, out=out, where=mask) def slash_where(data): a, b, mask = data return a / numpy.where(mask, b, 1.0) def numexpr_eval(data): a, b, mask = data return numexpr.evaluate('a / (1 - mask + b)') b = perfplot.bench( setup=setup, kernels=[copy_slash, copy_divide, slash_where, numexpr_eval], n_range=[2 ** k for k in range(24)], xlabel="n" ) b.save("out.png")
Question: What I Have Dictionary: user_dict = {'user': {'field1': 'value1', 'field2': 'value2'}, 'admin':{'field1': 'value3', 'field2': 'value4'}}  Pydantic Model: class User(BaseModel): account_type: Optional[str] = 'user' field1: Optional[str] = '' field1: Optional[str] = '' class Config: validate_assignment = True @validator("account_type", pre=True, always=True) def _set_account_type(cls, account_type: str): return account_type or "user"  Desired Outcome I would like to lookup the values of field1 and field2 from the user_dict based upon the account_type. So I should be able to do something like this: user = User() print(user.field1) # > 'value1' print(user.field2) # > 'value2' user2 = User(field1=None, field2=None) print(user2.field1) # > 'value1' print(user2.field2) # > 'value2' user3 = User(account_type="admin") print(user3.field1) # > 'value3' print(user3.field2) # > 'value4'  What I have tried class User(BaseModel): account_type: Optional[str] = 'user' field1: Optional[str] = user_dict['account_type']['field1'] field1: Optional[str] = user_dict['account_type']['field2'] class Config: validate_assignment = True @validator("account_type", pre=True, always=True) def _set_account_type(cls, account_type: str): return account_type or "user" user = User() print(user.field1) # > 'value1' user = User(field1=None) print(user.field1) # > None  Using Validators for field1 and field2: class User(BaseModel): account_type: Optional[str] = 'user' field1: Optional[str] field1: Optional[str] class Config: validate_assignment = True @validator("account_type", pre=True, always=True) def _set_account_type(cls, account_type: str): return account_type or "user" @validator("field1", pre=True, always=True) def _set_plan_credits(cls, value): if value is None: value = 'value1' return user_dict['account_type']['field1'] or value @validator("field2", pre=True, always=True) def _set_rollover_credits(cls, value): if value is None: value = 'value2' return user_dict['account_type']['field2'] or value user = User() # > AttributeError: type object 'Account' has no attribute 'account_type'  Any ideas on what I can do to get the desired outcome?

Answer: I solved it by using the root_validator decorator as follows: Solution: @root_validator(pre=False) def _set_fields(cls, values: dict) -> dict: """This is a validator that sets the field values based on the the user's account type. Args: values (dict): Stores the attributes of the User object. Returns: dict: The attributes of the user object with the user's fields. """ values["field1"] = user_dict[values["account_type"]]["field1"] values["field2"] = user_dict[values["account_type"]]["field2"] return values
Question: I'm trying to install Tensorflow Object Detection API, following the steps at this link, which is the official installation's documentation for Tensorflow 2. git clone https://github.com/tensorflow/models.git > everything is ok cd models/research/ > everything is ok protoc object_detection/protos/*.proto --python_out=. > everything is ok cp object_detection/packages/tf2/setup.py . > everything is ok python -m pip install --use-feature=2020-resolver . > Usage: > /opt/anaconda3/envs/ml/bin/python -m pip install [options] <requirement specifier> [package-> index-options] ... > /opt/anaconda3/envs/ml/bin/python -m pip install [options] -r <requirements file> [package-index-options] ... > /opt/anaconda3/envs/ml/bin/python -m pip install [options] [-e] <vcs project url> ... > /opt/anaconda3/envs/ml/bin/python -m pip install [options] [-e] <local project path> ... > /opt/anaconda3/envs/ml/bin/python -m pip install [options] <archive url/path> ... > no such option: --use-feature  Can someone help me understand why the installation stops as it does? I'm using macOS Mojave, Python 3.6 (on a conda virtual env), and Tensorflow 2.3.0.

Answer: I had the same problem, I upgraded pip version from 20.0.2 to 20.2.2, then it worked. An issue was opened on github on this matter, check here. Use python -m pip install --upgrade pip to upgrade pip.
Question: I want to transfer money between my coinbase accounts. I'm storing all of my accounts' IDs from client.get_accounts()['data']['id'] and transferring with the code, tx = client.transfer_money('2bbf394c-193b-5b2a-9155-3b4732659ede', to='58542935-67b5-56e1-a3f9-42686e07fa40', amount='1', currency= 'BTC)  But, I get this error. coinbase.wallet.error.APIError: APIError(id=):

Answer: I struggled with the same problem. It seems to be on their side and not limited to the python client. The only way I managed to transfer from wallet to wallet is by using the undocumented and unimplemented API "trades" that is used by the website. First your have to find the base_id of both your currencies, then your can do: r = client._post('v2', "trades", data={ "amount":"1", "amount_asset":"BTC", "amount_from":"input", "source_asset":"<BASE_ID_OF_SOUCE>", "target_asset":"<BASE_ID_OF_TARGET" } ) result = r.json() trade_id = result['data']['id'] client._post("v2", "trades", trade_id, "commit")  It's not the cleanest code since it's accessing a protected method and I'm not entirely sure that coinbase is OK with it (There might be a reason it's not documented...) but it does the job.