eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using Airflow's BigQueryOperator to populate the BQ table with write_disposition='WRITE_TRUNCATE'. The problem is that every time the task runs, it alters the table schema and also the column mode from Required to Nullable. The create_disposition I am using is 'CREATE_NEVER'. Since my tables are pre-created, I don't want the schemas or column modes to be altered. Using write_disposition='WRITE_APPEND' fixes the issue but my requirement is to use WRITE_TRUNCATE. Any idea why BigQueryOperator alters the schema and mode?

Answer: I had a similar issue, not with the required/nullable shcema value, but on policy tags, and the behavior is the same: the policy tags are overriden (and lost). Here the answer of the Google support team:  If you overwrite to a destination table, any existing policy tags are removed from the table, unless you use the --destination_schema flag to specify a schema with policy tags. For WRITE_TRUNCATE, the disposition overwrites the existing table and the schema. If you want to keep the policy tags, you can use "--destination_schema" to specify a schema with policy tags.  However, with my test in python, I observed 2 different behaviors between the QueryJob (job based on sql query and that sinks the result in a table) and the LoadJob (job that loads data from a file and that sinks the data in a table).  If you perform a LoadJob,  Remove the schema autodetec Get the schema of the original table Perform the load job    Like this in Python  job_config = bigquery.job.LoadJobConfig() job_config.create_disposition = bigquery.job.CreateDisposition.CREATE_IF_NEEDED job_config.write_disposition = bigquery.job.WriteDisposition.WRITE_TRUNCATE job_config.skip_leading_rows = 1 # job_config.autodetect = True job_config.schema = client.get_table(table).schema query_job = client.load_table_from_uri(uri, table, job_config=job_config) res = query_job.result()  This solution, to copy the schema, doesn't work with a QueryJob  The workaround is the following (works for LoadJob and QueryJob)  Truncate the table Perform a job in WRITE_EMPTY mode  Trade-off:  the WRITE_TRUNCATE is atomic: if the write write failed, the data aren't truncated the workaround is in 2 steps: if the write write failed, the data are already deleted   config = bigquery.job.QueryJobConfig() config.create_disposition = bigquery.job.CreateDisposition.CREATE_IF_NEEDED config.write_disposition = bigquery.job.WriteDisposition.WRITE_EMPTY # Don't work # config.schema = client.get_table(table).schema config.destination = table # Step 1 truncate the table query_job = client.query(f'TRUNCATE TABLE `{table}`') res = query_job.result() # Step 2: Load the new data query_job = client.query(request, job_config=job_config) res = query_job.result()   All of this to tell you that the BigQuery operator on Airflow isn't the problem. It's a BigQuery issue. You have a workaround to achieve what you want.
Question: I have a plotly graph with subplots drawn this way: fig = make_subplots( rows=4, cols=1, subplot_titles=("Price, orders and positions", "Margin use", "PnL and fees", "Volume traded"), row_heights=[0.5, 0.2, 0.2, 0.1], vertical_spacing=0.1 ) # price, orders, etc fig.add_traces( [ # draw price, average price and min / max go.Scatter(name='Price', x=df.index, y=df['price'], mode='lines', line=dict(color='rgba(31, 119, 180, 1.)')), go.Scatter(name='Average Price', x=df.index, y=df['average_price'], mode='lines', line=dict(color='rgba(31, 119, 180, 0.5)')), go.Scatter(x=df.index, y=df['price_max'], mode='lines', marker=dict(color="#444"), line=dict(width=0), showlegend=False), go.Scatter(x=df.index, y=df['price_min'], marker=dict(color="#444"), line=dict(width=0), mode='lines', fillcolor='rgba(68, 68, 68, 0.3)', fill='tonexty', showlegend=False), # draw the long / short orders go.Scatter(name='Long Open Price', x=df.index, y=df['orderlo_price'], mode='lines', line=dict(color='rgb(180, 119, 31)')), go.Scatter(name='Long Close Price', x=df.index, y=df['orderlc_price'], mode='lines', line=dict(width=2, color='rgb(220, 159, 31)')), go.Scatter(name='Short Open Price', x=df.index, y=df['orderso_price'], mode='lines', line=dict(color='rgb(119, 180, 31)')), go.Scatter(name='Short Close Price', x=df.index, y=df['ordersc_price'], mode='lines', line=dict(width=2, color='rgb(159, 220, 31)')), # add the position go.Scatter(name='position', x=df.index, y=df['position_price'], mode='lines', line=dict(color='rgb(240, 200, 40)')) ], rows=[1, 1, 1, 1, 1, 1, 1, 1, 1], cols=[1, 1, 1, 1, 1, 1, 1, 1, 1] ) # margin use fig.add_traces( [ go.Scatter(name='Margin use', x=df.index, y=df['margin_use'], mode='lines', line=dict(color='rgba(31, 119, 180, 1.)')), #go.Scatter(name='Margin max', x=df.index, y=df['margin_max'], mode='lines', line=dict(color='rgba(31, 180, 119, 1.)')) ], rows=[2], #, 2], cols=[1], #, 1] ) # PnL and fees fig.add_traces( [ go.Scatter(name='Unrealized PnL', x=df.index, y=df['unrealized_pnl'], line=dict(color='rgb(64, 255, 64, 1.0)')), go.Scatter(name='Realized PnL', x=df.index, y=df['realized_pnl'], fill='tozeroy', line=dict(color='rgb(32, 192, 32, 0.8)')), go.Scatter(name='Fees Paid', x=df.index, y=df['fees_paid'], fill='tozeroy', line=dict(color='rgb(192, 32, 32, 0.4)')) ], rows=[3, 3, 3], cols=[1, 1, 1] ) # Volume traded fig.add_trace( go.Scatter(name='Volume Traded', x=df.index, y=df['volume_traded'], fill='tozeroy', line=dict(color='rgb(32, 128, 192, 1.0)')), row=4, col=1 ) # remove the margins fig.update( layout=go.Layout( margin=go.layout.Margin(l=0.5, r=0.5, b=0.5, t=50) ) )  how can I make sure all the subplots are synchronized regarding the X axis zoom and panning? Right now you can zoom in one subplot and suddenly the graph in that subplot has no relations to the other subplots.

Answer: Have a look at the Shared X-Axes section of the Plotly docs. I believe this is what you're looking for. Essentially, add shared_xaxes=True to the make_subplots() function.
Question: I have survey data which annoying has returned multiple choice questions in the following way. It's in an excel sheet There is about 60 columns with responses from single to multiple that are split by /. This is what I have so far, is there any way to do this quicker without having to do this for each individual column data = {'q1': ['one', 'two', 'three'], 'q2' : ['one/two/three', 'a/b/c', 'd/e/f'], 'q3' : ['a/b/c', 'd/e/f','g/h/i']} df = pd.DataFrame(data) df[['q2a', 'q2b', 'q2c']]= df['q2'].str.split('/', expand = True, n=0) df[['q3a', 'q3b', 'q3c']]= df['q2'].str.split('/', expand = True, n=0) clean_df = df.drop(df[['q2', 'q3']], axis=1) 

Answer: We can use list comprehension with add_prefix, then we use pd.concat to concatenate everything to your final df: splits = [df[col].str.split(pat='/', expand=True).add_prefix(col) for col in df.columns] clean_df = pd.concat(splits, axis=1)   q10 q20 q21 q22 q30 q31 q32 0 one one two three a b c 1 two a b c d e f 2 three d e f g h i   If you actually want your column names to be suffixed by a letter, you can do the following with string.ascii_lowercase: from string import ascii_lowercase dfs = [] for col in df.columns: d = df[col].str.split('/', expand=True) c = d.shape[1] d.columns = [col + l for l in ascii_lowercase[:c]] dfs.append(d) clean_df = pd.concat(dfs, axis=1)   q1a q2a q2b q2c q3a q3b q3c 0 one one two three a b c 1 two a b c d e f 2 three d e f g h i
Question: I have a file in python like: def test_constructor_for_legacy_json(): """Test if constructor works for a legacy JSON in an old database""" a = A(**{ 'field1': 'BIG TEXT WITH MORE THAN 500 CHARACTERS....(...)', 'field2': 'BIG TEXT WITH MORE THAN 500 CHARACTERS....(...)', 'field3': 'BIG TEXT WITH MORE THAN 500 CHARACTERS....(...)', # (...) 'field1000': 'BIG TEXT WITH MORE THAN 500 CHARACTERS....(...)', }) assert type(a) == A  When I run flake8 + hacking I receive an error because the lines are too big. If I put this command at the beginning of the file # flake8: noqa all file will be ignored from linter. But I only want to exclude from linter the block where a is declared. I want to lint the rest of the file, and I cannot put at the end of each fieldx an # noqa: E501. Some one know how can I solve this? Thanks

Answer: There isn't a way in flake8 to ignore a block of code Your options are:  ignore each line that produces an error by putting # noqa: E501 on it  ignore the entire file (but this turns off all other errors as well) with a # flake8: noqa on a line by itself  ignore E501 in the entire file by using per-file-ignores: [flake8] per-file-ignores = path/to/file.py: E501    generally I'd prefer the third one, maybe even sequestering your long-strings into their own file to be ignored  disclaimer: I'm the current flake8 maintainer
Question: I'm trying to identify the parameters of a function for which default values are not set. I'm using inspect.signature(func).parameters.value() function which gives a list of function parameters. Since I'm using PyCharm, I can see that the parameters for which the default value is not set have their Parameter.default attribute set to inspect._empty. I'm declaring the function in the following way: def f(a, b=1): pass  So, the default value of a is inspect._empty. Since inspect._empty is a private attribute, I thought that there might be a method for checking if a value is inspect._empty, but I couldn't find it.

Answer: You can do it like this: import inspect def foo(a, b=1): pass for param in inspect.signature(foo).parameters.values(): if param.default is param.empty: print(param.name)  Output: a  param.empty holds the same object inspect._empty. I suppose that this way of using it is recommended because of the example in the official documentation of inspect module: Example: print all keyword-only arguments without default values: >>> >>> def foo(a, b, *, c, d=10): ... pass >>> sig = signature(foo) >>> for param in sig.parameters.values(): ... if (param.kind == param.KEYWORD_ONLY and ... param.default is param.empty): ... print('Parameter:', param) Parameter: c
Question: I have a dataclass, which looks like this: @dataclass class myClass: id: str mode: str value: float  This results in: dataclasses.asdict(myClass) {"id": id, "mode": mode, "value": value}  But what I want is {id:{"mode": mode, "value": value}}  I thought I could achive this by adding a to_dict method to my dataclass, which returns the desired dict, but that didn't work. How could I get my desired result?

Answer: from dataclasses import dataclass, asdict @dataclass class myClass: id: str mode: str value: float def my_dict(data): return { data[0][1]: { field: value for field, value in data[1:] } } instance = myClass("123", "read", 1.23) data = {"123": {"mode": "read", "value": 1.23}} assert asdict(instance, dict_factory=my_dict) == data
Question: I would like to select those rows in a column that contains any of the substrings in a list. This is what I have for now. product = ['LID', 'TABLEWARE', 'CUP', 'COVER', 'CONTAINER', 'PACKAGING'] df_plastic_prod = df_plastic[df_plastic['Goods Shipped'].str.contains(product)] df_plastic_prod.info()  Sample df_plastic Name Product David PLASTIC BOTTLE Meghan PLASTIC COVER Melanie PLASTIC CUP Aaron PLASTIC BOWL Venus PLASTIC KNIFE Abigail PLASTIC CONTAINER Sophia PLASTIC LID  Desired df_plastic_prod Name Product Meghan PLASTIC COVER Melanie PLASTIC CUP Abigail PLASTIC CONTAINER Sophia PLASTIC LID  Thanks in advance! I appreciate any assistance on this!

Answer: For match values by subtrings join all values of list by | for regex or - so get values LID or TABLEWARE ...: Solution working well also with 2 or more words in list. pat = '|'.join(r"\b{}\b".format(x) for x in product) df_plastic_prod = df_plastic[df_plastic['Product'].str.contains(pat)] print (df_plastic_prod) Name Product 1 Meghan PLASTIC COVER 2 Melanie PLASTIC CUP 5 Abigail PLASTIC CONTAINER 6 Sophia PLASTIC LID
Question: import cv2 as cv import numpy as np from matplotlib import pyplot as plt img = cv.imread('t2.jpg', cv.COLOR_BGR2GRAY) blur = cv.GaussianBlur(img, (5, 5), 0) ret3, th3 = cv.threshold(blur, 0, 255, cv.THRESH_BINARY + cv.THRESH_OTSU)  I get the error: Traceback (most recent call last): File "F:/l4 project docs/project/l4proTest1/projectionStepTest.py", line 11, in <module> ret3,th3 = cv.threshold(blur, 0, 255, cv.THRESH_BINARY + cv.THRESH_OTSU) cv2.error: OpenCV(4.4.0) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-9d_dfo3_\opencv\modules\imgproc\src\thresh.cpp:1557: error: (-2:Unspecified error) in function 'double __cdecl cv::threshold(const class cv::_InputArray &,const class cv::_OutputArray &,double,double,int)'  THRESH_OTSU mode: 'src_type == CV_8UC1 || src_type == CV_16UC1' where 'src_type' is 16 (CV_8UC3) Process finished with exit code 1 

Answer: img = cv.imread('t2.jpg', cv.IMREAD_GRAYSCALE)  or img = cv.imread('t2.jpg', cv.IMREAD_COLOR) img = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
Question: I am writing a Telegram bot using the PyTelegramBotApi library, I would like to implement the function of checking the user's subscription to a certain telegram channel, and if there is none, offer to subscribe. Thanks in advance for your answers!

Answer: use getChatMember method to check if a user is member in a channel or not.  getChatMember Use this method to get information about a member of a chat. Returns a ChatMember object on success.  import telebot bot = telebot.TeleBot("TOKEN") CHAT_ID = -1001... USER_ID = 700... result = bot.get_chat_member(CHAT_ID, USER_ID) print(result) bot.polling()  Sample result: You receive a user info if the user is a member {'user': {'id': 700..., 'is_bot': False, 'first_name': '', 'username': None, 'last_name': None, ... }  or an Exception otherwise telebot.apihelper.ApiTelegramException: A request to the Telegram API was unsuccessful. Error code: 400 Description: Bad Request: user not found   example on how to use it inside your project import telebot from telebot.apihelper import ApiTelegramException bot = telebot.TeleBot("BOT_TOKEN") CHAT_ID = -1001... USER_ID = 700... def is_subscribed(chat_id, user_id): try: bot.get_chat_member(chat_id, user_id) return True except ApiTelegramException as e: if e.result_json['description'] == 'Bad Request: user not found': return False if not is_subscribed(CHAT_ID, USER_ID): # user is not subscribed. send message to the user bot.send_message(CHAT_ID, 'Please subscribe to the channel') else: # user is subscribed. continue with the rest of the logic # ... bot.polling()
Question: I can't figure out how to avoid this doctest error: Failed example: print(test()) Expected: output <BLANKLINE> Got: output <BLANKLINE>  For this code def test(): r'''Produce string according to specification. >>> print(test()) output <BLANKLINE> ''' return '\toutput\n'  I have put a tab literal into the source code, line 5 in front of output. It looks like doctest (or python docstrings?) ignores that tab literal and converts it to four spaces. The so-called "Expected" value is literally not what my source specifies it to be. What's a solution for this? I don't want to replace the print statement with >>> test() '\toutput\n'  Because a huge part of why I usually like doctests is because they demonstrate examples, and the most important part of this function I am writing is the shape of the output.

Answer: The tabs in the docstring get expanded to 8 spaces, but the tabs in the output are not expanded. From the doctest documentation (emphasis added):  All hard tab characters are expanded to spaces, using 8-column tab stops. Tabs in output generated by the tested code are not modified. Because any hard tabs in the sample output are expanded, this means that if the code output includes hard tabs, the only way the doctest can pass is if the NORMALIZE_WHITESPACE option or directive is in effect. Alternatively, the test can be rewritten to capture the output and compare it to an expected value as part of the test. This handling of tabs in the source was arrived at through trial and error, and has proven to be the least error prone way of handling them. It is possible to use a different algorithm for handling tabs by writing a custom DocTestParser class.  Unfortunately, all examples of directive use in the doctest docs have been mangled by Sphinx. Here's how you would use NORMALIZE_WHITESPACE: def test(): r'''Produce string according to specification. >>> print(test()) # doctest: +NORMALIZE_WHITESPACE output <BLANKLINE> ''' return '\toutput\n'  Note that this treats all runs of whitespace as equal, rather than disabling the tab processing. Disabling the tab processing is absurdly cumbersome, and it's impossible through the usual doctest interface - you would need to subclass DocTestParser, manually reimplement doctest parsing, then construct instances of DocTestFinder, DocTestRunner, and your DocTestParser subclass and call their methods manually.
Question: I'm trying to find the angle between two vectors. Following is the code that I use to evaluate the angle between vectors ba and bc import numpy as np import scipy.linalg as la a = np.array([6,0]) b = np.array([0,0]) c = np.array([1,1]) ba = a - b bc = c - b cosine_angle = np.dot(ba, bc) / (la.norm(ba) * la.norm(bc)) angle = np.arccos(cosine_angle) print (np.degrees(angle))  My question is, here in this code: for both c = np.array([1,1]) and c = np.array([1,-1]) you get 45 degrees as the answer. I can understand this in a mathematical viewpoint because, from the dot product you always focus on the angle in the interval [0,180]. But geometrically this is misleading as the point c is in two different locations for [1,1] and [1,-1]. So is there a way that I can get the angle in the interval [0,360] for a general starting point b = np.array([x,y]) Appreciate your help

Answer: Conceptually, obtaining the angle between two vectors using the dot product is perfectly alright. However, since the angle between two vectors is invariant upon translation/rotation of the coordinate system, we can find the angle subtended by each vector to the positive direction of the x-axis and subtract one value from the other. The advantage is, we'll use np.arctan2to find the angles, which returns angles in the range [-π,π] and hence you get an idea of the quadrant your vector lies in. # Syntax: np.arctan2(y, x) - put the y value first! # Instead of explicitly referring by indices, you can unpack each vector in reverse, like so: # np.arctan2(*bc[::-1]) angle = np.arctan2(bc[1], bc[0]) - np.arctan2(ba[1], ba[0])  Which you can then appropriately transform to get a value within [0, 2π].
Question: I'm running pytest with the coverage plugin (pytest --cov) and in the report I got the following line: Name Stmts Miss Branch BrPart Cover Missing --------------------------------------------------------- foo.py 5 1 2 1 71% 3->5, 5  I know that 3-5 would mean it missed lines 3 to 5, but I don't know what -> means. From the test logic, I'd expect only 5 to be reported. For reference, this is the code I used: # foo.py class Test: def __lt__(self, other): if type(self) == type(other): return False return NotImplemented # test_foo.py def test_lt(): test = Test() assert not (test < test) 

Answer: Coverage collects pairs of transition in your code that goes from one line (the source) to another (the destination). In some cases, some transitions could be jumped, like in conditional statements or a break statememt, then it would be measured as a missing branch (or missing transition). For example, in your code there is one transition that can be jumped. if type(self) == type(other): return False return NotImplemented  see that from line 3 to line 5 is a transition that can not necessarily happen as there could be the case that the if statement don't evaluate to False. Thus, branch coverage will flag this code as not fully covered because of the missing jump from line 3 to line 5. References How does branch coverage works. https://coverage.readthedocs.io/en/latest/branch.html#how-it-works
Question: I have found examples of how to remove a column based on all or a threshold but I have not been able to find a solution to my particular problem which is dropping the column if the last row is nan. The reason for this is im using time series data in which the collection of data doesnt all start at the same time which is fine but if I used one of the previous solutions it would remove 95% of the dataset. I do however not want data whose most recent column is nan as it means its defunct. A B C nan t x 1 2 3 x y z 4 nan 6  Returns A C nan x 1 3 x z 4 6 

Answer: You can also do something like this df.loc[:, ~df.iloc[-1].isna()]   A C 0 NaN x 1 1 3 2 x z 3 4 6
Question: I'm trying to make a lexer in python but when I try to import a class from file token.py I get this error ImportError: cannot import name 'Token' from 'token'  the code for token.py is as follows from enum import Enum class Token(): def __init__(self, ttype, value=None): self.type = ttype self.value = value def __repr__(self): return {'type':self.type, 'value':self.value} class TokenType(Enum): NUMBER = 0 PLUS = 1 MINUS = 2 MULTIPLY = 3 DIVIDE = 4 LPAREN = 5 RPAREN = 6  and the import statement is from token import Token, TokenType 

Answer: There is a library in python called token, so your interpreter might be confusing it with the inbuilt python library. Try to rename the library. Name it token_2.py or something
Question: The test.csv data likes this: device_id,upload_time,latitude,longitude,mileage,other_vals,speed,upload_time_1 11115304371,2020-08-05 05:10:05+00:00,23.140366,114.18685,0,,0,202008 1234,2020-08-05 05:10:33+00:00,22.994716,114.2998,0,,0,202008 11115304371,2020-08-05 05:20:55+00:00,22.994716,114.2998,0,,3.8,202008 11115304371,2020-08-05 05:24:02+00:00,22.994916,114.299683,0,,2.1,202008 11115304371,2020-08-05 05:24:30+00:00,22.99545,114.2998,0,,6.5,202008 11115304371,2020-08-05 05:29:30+00:00,22.995433,114.299766,0,,3.4,202008 11115304371,2020-08-05 05:34:30+00:00,22.995433,114.299766,0,,3.4,202008 11115304371,2020-08-05 05:39:30+00:00,22.995433,114.299766,0,,3.4,202008 822649e2d142a486,2020-08-05 05:44:30+00:00,22.995433,114.299766,0,,3.4,202008 11115304371,2020-08-05 05:44:53+00:00,22.995433,114.299766,0,,3.4,202008 11115304371,2020-08-05 05:45:40+00:00,22.995433,114.299766,0,,5.8,202008  And the info.csv data likes this: car_id,device_id,car_type,car_num,marketer_name 1,11110110037,1,AAA,T1 2,11115304371,1,BBB,T2 3,11111100345,1,CCC,T3 4,11111100242,1,DDD,T4 5,12221100034,1,EEE,T5 6,12221100230,1,FFF,T6 7,14465301234,1,GGG,T7  When I use this code to merge 2 dataframe. import pandas as pd df_device_data = pd.read_csv(r'E:/test.csv', encoding='utf-8', parse_dates=[1], low_memory=False) df_common_car_info = pd.read_csv(r'E:/info.csv', encoding='utf-8', low_memory=False) result = pd.merge(df_device_data, df_common_car_info, how='left', on='device_id') result.to_csv(r'E:/result.csv', index=False, mode='w', header=True)  This error occurred:  ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat  How to fix it?

Answer: When I use this code df.astype(str): import pandas as pd df_device_data = pd.read_csv(r'E:/test.csv', encoding='utf-8', parse_dates=[1], low_memory=False) df_device_data['device_id'] = df_device_data['device_id'].astype(str) df_common_car_info = pd.read_csv(r'E:/info.csv', encoding='utf-8', low_memory=False) df_common_car_info['device_id'] = df_common_car_info['device_id'].astype(str) result = pd.merge(df_device_data, df_common_car_info, how='left', on='device_id') result.to_csv(r'E:/result.csv', index=False, mode='w', header=True)  The result is right.