eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I occasionally use the "trick" to extend a list by a mapped version of itself, for example to efficiently compute powers of 2: from operator import mul powers = [1] powers += map(mul, [2] * 10, powers) print(powers) # prints [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]  This relies on the += immediately appending each value from map to the list, so that the map then finds it and the procedure continues. In other words, it needs to work like this: powers = [1] for value in map(mul, [2] * 10, powers): powers.append(value)  And not first compute and store the whole right-hand side like this, where powers ends up being [1, 2]: powers = [1] powers += list(map(mul, [2] * 10, powers))  Is it somewhere guaranteed that it works like it does? I checked the Mutable Sequence Types documentation but it doesn't say much about it other than implying equivalence of s += t and s.extend(t). It does refer to MutableSequence, whose source code includes this:  def extend(self, values): 'S.extend(iterable) -- extend sequence by appending elements from the iterable' if values is self: values = list(values) for v in values: self.append(v)   def __iadd__(self, values): self.extend(values) return self  This does suggest that it's indeed supposed to work like it does and like I want it, but some source code being what it is doesn't feel as safe as a guarantee in the documentation.

Answer: I don't see any tests or docs that the greedy behavior is guaranteed; however, I do think it is the expected behavior and that code in the wild relies on it. FWIW, += with lists is equivalent to list.extend(), so your "trick" boils down to: >>> powers = [1] >>> powers.extend(2*x for x in islice(powers, 10)) >>> powers [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]  While I haven't found a guarantee for += or extend, we do have a guarantee that the list iterator allows mutation while iterating.¹ So, this code is on firm ground: >>> powers = [1] >>> for x in powers: if len(powers) == 10: break powers.append(2 * x) >>> powers [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]  ¹ See the second paragraph following the table at: https://docs.python.org/3/library/stdtypes.html#common-sequence-operations:  Forward and reversed iterators over mutable sequences access values using an index. That index will continue to march forward (or backward) even if the underlying sequence is mutated. The iterator terminates only when an IndexError or a StopIteration is encountered (or when the index drops below zero).
Question: For example when I give 5 to the code, I want to turn on the LED in our RPi pico (connected to a PC via a cable). #This code will run in my computer (test.py) x=int(input("Number?")) if (x==5): #turn on raspberry pi pico led  The code of the RPi pico: #This code will run in my rpi pico (pico.py) from machine import Pin led = Pin(25, Pin.OUT) led.value(1)  Or vice versa (doing something in the code on the computer with the code in the RPi pico). How can I call/get a variable in the PC to the RPi pico? Note: I am writing code with OpenCV Python and I want to process the data from my computer's camera on my computer. I want the RPi pico to react according to the processed data.

Answer: A simple method of communicating between the host and the Pico is to use the serial port. I have a rp2040-zero, which presents itself to the host as /dev/ttyACM0. If I use code like this on the rp2040: import sys import machine led = machine.Pin(24, machine.Pin.OUT) def led_on(): led(1) def led_off(): led(0) while True: # read a command from the host v = sys.stdin.readline().strip() # perform the requested action if v.lower() == "on": led_on() elif v.lower() == "off": led_off()  Then I can run this on the host to blink the LED: import serial import time # open a serial connection s = serial.Serial("/dev/ttyACM0", 115200) # blink the led while True: s.write(b"on\n") time.sleep(1) s.write(b"off\n") time.sleep(1)  This is obviously just one-way communication, but you could of course implement a mechanism for passing information back to the host.
Question: I was wondering whether x**2 or x*x is faster def sqr(x): for i in range (20): x = x**2 return x def sqr_(x): for i in range (20): x = x*x return x  When I time it, this is what I get: The time it takes for x**2: 101230500 The time it takes for x*x: 201469200  I have tried it many many times, they are either equal, or x ** 2 is faster than x * x. But x*x is never faster than x**2. So I disassembled the code: For x**2:  5 12 LOAD_FAST 0 (x) 14 LOAD_CONST 2 (2) 16 BINARY_POWER 18 STORE_FAST 0 (x) 20 JUMP_ABSOLUTE 8  For x*x:  9 12 LOAD_FAST 0 (x) 14 LOAD_FAST 0 (x) 16 BINARY_MULTIPLY 18 STORE_FAST 0 (x) 20 JUMP_ABSOLUTE 8  Is it about load_const being slightly faster than load_fast?  LOAD_CONST: takes the literal value at index 1 of co_consts and pushes it   LOAD_FAST is accessing the value in an array by index  Or binary_power is faster than binary_multiply (I actually don't know the binary_power algorithm)?

Answer: For small integers, x*x is significantly faster than x**2 since CPython does a lot more operation internally to compute a**b. Actually, on my machine x*x is 4 times faster (processor i5-9600KF, CPython 3.8.1, on Windows). That being said, in you code, numbers grows very quickly and Python integers are unbounded. In fact, each exponentiation cause the binary representation to be twice bigger. The exponents are multiplied together resulting in the computation of x**(2*2*2*...*2) = x**(2**20) = x**1048576. For big x=2, the number takes 128 KiB in memory and for x=100 it takes 850 KiB. This is pretty big. Each iteration of your loop is bounded by the computation of such huge numbers in memory. As a result, for large numbers, x*x and x**2 are as fast because the same internal computation is done for both cases and the overhead of the CPython interpreter becomes negligible compared to the computation of the huge integers.  Under the hood Internally, CPython appears to use _PyNumber_PowerNoMod which calls PyNumber_Power which calls ternary_op, and PyNumber_Multiply which calls binary_op1. Note that CPython is not optimized to compute x**2: internally CPython compute pow(x, 2, None) which is the function to compute a modular exponentiation (though the call to pow is a bit less efficient way as CPython has to check pow has not been overwritten). This modular exponentiation function is much more expensive since it is a very generic function compared to x * x. In the end, it appears long_mul and long_pow are called in your case (note that long_pow calls long_mul internally so long_pow actually need to compute more instructions). For large numbers, CPython uses the Karatsuba multiplication (see: k_mul). Note that CPython is actually very slow in both cases since it takes several nanoseconds (at least on my machine) and performs dozens of checks and many function calls just to multiply two integers. This can be done natively in only 1 cycle for 64-bit integers on mainstream x86-64 processors. Large integers cannot be natively computed by mainstream processor and require a much more expensive computation.
Question: There are a number of use cases for which I am trying to use conda . The main headache is that conda init just does not want to play fair within the flow of a script in the same bash shell. I frequently see  CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.  That happens even though the contents of the conda initialize script as well as conda init have been executed. The results are inconsistent: sometimes I have seen the init logic work, others not. I have not been able to ascertain what magic is happening and what it expects in order to work properly.. conda_init() { # __conda_setup="$('$CONDA_DIR/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"; source $CONDA_DIR/etc/profile.d/conda.sh export PATH="$CONDA_DIR/condabin:$CONDA_DIR/bin:$PATH"; export CONDARC=$CONDA_DIR ; conda init bash } conda_init conda activate py38  That gives us  /Users/steve/opt/miniconda3/bin/conda /Users/steve/opt/miniconda3/bin::/Users/steve/opt/miniconda3/condabin:<other stuff..> CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run $ conda init <SHELL_NAME> Currently supported shells are: - bash - fish - tcsh - xonsh - zsh - powershell See 'conda init --help' for more information and options. IMPORTANT: You may need to close and restart your shell after running 'conda init'.  How can that conda_init() be made reliable for being able to run conda init in the same shell?

Answer: Generally, one should not need to "activate" an environment when working programmatically. That's what conda run is for... conda run -n py38 python my_script.py  Otherwise, if CONDA_DIR is defined, then the following would run the initialization shell commands in an active bash session: eval "$(${CONDA_DIR}/bin/conda shell.bash hook 2> /dev/null)"
Question: How to fix TikTok selenium robot detection Background-Info  I'm creating a python selenium bot to do things on the TikTok website. The user will log in manually so the website detecting mouse movement and typing speed is irrelevant.The issue is, is that I can't log in while using selenium What I've tried   I've tried logging in normally without selenium in incognito mode on chrome with the same Mac address, IP address, and same login details (Which worked!!)  I've tried using random user agents in selenium (Which didn't Work)  I've tried adding the following chrome options   options.add_argument("start-maximized") # Chrome is controlled by automated test software options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option('useAutomationExtension', False) # avoiding detection options.add_argument('--disable-blink-features=AutomationControlled')   What I want  I want to be able to log in without TikTok saying Too many log-in attempts. Try again later and for more clarification, I can log in normally without selenium same everything and it works it just doesn't work while in selenium.  Heres is the code for starting selenium post = "https://www.tiktok.com/@smoothmovesranch/video/7091224442243681579" myProxy = "" #configuration options = Options() prox = Proxy() prox.proxy_type = ProxyType.MANUAL prox.http_proxy = myProxy prox.ssl_proxy = myProxy capabilities = webdriver.DesiredCapabilities.CHROME prox.add_to_capabilities(capabilities) options.add_argument("window-size=1400,600") options.add_argument("--incognito") driver = webdriver.Chrome(executable_path = os.path.join(os.getcwd(), 'chromedriver'), options=options) #opens tiktok login page driver.get('https://www.tiktok.com/login/phone-or-email/email') 

Answer: A few things that might help:  Make sure your proxy is changing during every login attempt.  For every instance of a new login create a new webdriver environment either with the same proxy or a new one.  Add random wait times. For example instagram will restrict accounts that they suspect of botting. To fix this one solution is to make the selenium instance preform different clicking actions at different times. i.e. having a wait time fluctuating between a few seconds can do the trick.  Also this code may help with too many login attempt issues. In short it helps selenium disguise itself better to the website servers when navigating the site.  user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36' options.add_argument('user-agent={0}'.format(user_agent))
Question: We have data as below  Name value1 Value2 finallist 0 cosmos 10 20 [10,20] 1 network 30 40 [30,40] 2 unab 20 40 [20,40]  is there any way to do difference between all the rows Something final output like  Name value1 Value2 finallist cosmos network unab 0 cosmos 10 20 [10,20] 0 40 30 1 network 30 40 [30,40] 40 0 10 2 unab 20 40 [20,40] 30 10 0  Data has different types of names and each name should be a column

Answer: You want the pairwise absolute difference of the sum of the values for each row. The easiest might be to use the underlying numpy array. absolute difference of the sum of the "value" columns # get sum of values per row and convert to numpy array a = df['value1'].filter(regex='(?i)value').sum(1).to_numpy() # compute the pairwise difference, create a DataFrame and join df2 = df.join(pd.DataFrame(abs(a-a[:,None]), columns=df['Name'], index=df.index))  output:  Name value1 Value2 finallist cosmos network unab 0 cosmos 10 20 [10, 20] 0 40 30 1 network 30 40 [30, 40] 40 0 10 2 unab 20 40 [20, 40] 30 10 0
Question: I have df going like pd.DataFrame([["A1" "B1", "C1", "P"], ["A2" "B2", "C2", "P"], ["A3" "B3", "C3", "P"]], columns=["col_a" "col_b", "col_c", "col_d"]) col_a col_b col_c col_d A1 B1 C1 P A2 B2 C2 P A3 B3 C3 P ...  the result I need is basically repeat and ensure that columns have P Q R extension in col_d for every unique row occurence col_a col_b col_c col_d A1 B1 C1 P A1 B1 C1 Q A1 B1 C1 R A2 B2 C2 P A2 B2 C2 Q A2 B2 C2 R A3 B3 C3 P A3 B3 C3 Q A3 B3 C3 R ...  All I have so far is: new_df = pd.DataFrame(np.repeat(df.values, 3, axis=0), columns=df.columns)  Which result in duplication of those values, but col_d is unchanged EDIT: Now I stumbled upon another need, where for every unique col_a and col_b I need to add "S" to col_d Resulting for instance in this: col_a col_b col_c col_d A1 B1 C1 P A1 B1 C1 Q A1 B1 C1 R A1 B1 T S A2 B2 C2 P A2 B2 C2 Q A2 B2 C2 R A2 B2 T S  Thank you very much for help!

Answer: Add values to column col_d by DataFrame.assign with numpy.tile: L = ['P','Q','R'] new_df = (pd.DataFrame(np.repeat(df.values, 3, axis=0), columns=df.columns) .assign(col_d = np.tile(L, len(df)))) print (new_df) col_acol_b col_c col_d 0 A1B1 C1 P 1 A1B1 C1 Q 2 A1B1 C1 R 3 A2B2 C2 P 4 A2B2 C2 Q 5 A2B2 C2 R 6 A3B3 C3 P 7 A3B3 C3 Q 8 A3B3 C3 R  Another similar idea is repeat indices and duplicated rows by DataFrame.loc: L = ['P','Q','R'] new_df = (df.loc[df.index.repeat(3)] .assign(col_d = np.tile(L, len(df))) .reset_index(drop=True)) print (new_df) col_acol_b col_c col_d 0 A1B1 C1 P 1 A1B1 C1 Q 2 A1B1 C1 R 3 A2B2 C2 P 4 A2B2 C2 Q 5 A2B2 C2 R 6 A3B3 C3 P 7 A3B3 C3 Q 8 A3B3 C3 R  EDIT: L = ['P','Q','R','S'] new_df = (pd.DataFrame(np.repeat(df.values, len(L), axis=0), columns=df.columns) .assign(col_d = np.tile(L, len(df)), col_c = lambda x: x['col_c'].mask(x['col_d'].eq('S'), 'T'))) print (new_df) col_acol_b col_c col_d 0 A1B1 C1 P 1 A1B1 C1 Q 2 A1B1 C1 R 3 A1B1 T S 4 A2B2 C2 P 5 A2B2 C2 Q 6 A2B2 C2 R 7 A2B2 T S 8 A3B3 C3 P 9 A3B3 C3 Q 10 A3B3 C3 R 11 A3B3 T S
Question: I have two tasks inside a TaskGroup that need to pull xcom values to supply the job_flow_id and step_id. Here's the code:  with TaskGroup('execute_my_steps') as execute_my_steps: config = {some dictionary} dependencies = {another dictionary} task_id = 'execute_spark_job_step' task_name = 'spark_job' add_step = EmrAddStepsOperator( task_id=task_id, job_flow_id="{{ task_instance.xcom_pull(dag_id='my_dag', task_ids='emr', key='return_value') }}", steps=create_emr_step(args=config, d=dependencies), aws_conn_id='aws_default', retries=3, dag=dag ) wait_for_step = EmrStepSensor( task_id='wait_for_' + task_name + '_step', job_flow_id="{{ task_instance.xcom_pull(dag_id='my_dag', task_ids='emr', key='return_value') }}", step_id="{{ task_instance.xcom_pull(dag_id='my_dag', task_ids='" + task_id + "', key='return_value') }}", retries=3, dag=dag, mode='reschedule' ) add_step >> wait_for_step  The problem is the step_id does not render correctly. The wait_for_step value in the UI rendered template shows as 'None', however, the xcom return_value for execute_spark_job_step is there (this is the emr step_id). wait_for_step rendered template:  execute_spark_job_step xcom:  When I remove the TaskGroup, it renders fine and the step waits until the job enters the completed state. I need this to be in a task group because I will be looping through a larger config file and creating multiple steps. Why doesn't this work? Do I need a nested TaskGroup? I tried using a TaskGroup without the context manager and still no luck.

Answer: TL;DR: Your issue is happening because the id is not task_id it's group_id.task_id so your code should be: task_ids=f"execute_my_steps.{ task_id }"  =>  step_id="{{ task_instance.xcom_pull(dag_id='my_dag', task_ids=f"execute_my_steps.{ task_id }", key='return_value') }}",  The explanation why it happens: When task is assigned to TaskGroup the id of the task is no longer the task_id but it becomes group_id.task_id to reflect this relationship. In Airflow task_id is unique but when you use TaskGroup you can set the same task_id in different TaskGroups. If this behavior is not something that you want, you can disable it by setting prefix_group_id=False in your TaskGroup: with TaskGroup( group_id='execute_my_steps', prefix_group_id=False ) as execute_my_steps:  By doing so your code will work without changes. The task_id will simply be task_id without the group_id prefix. Note that this also means that it's up to you to make sure you don't have duplicated task_ids in your DAG.
Question: Suppose I have a simple dataframe where I have four features as food, kitchen, city, and detail. d = {'Food': ['P1|0', 'P2', 'P3|45', 'P1', 'P2', 'P4', 'P1|1', 'P3|7', 'P5', 'P1||23'], 'Kitchen' : ['L1', 'L2','L9', 'L4','L5', 'L6','L1', 'L9','L10', 'L1'], 'City': ['A', 'A', 'A', 'B', 'B','B', 'C', 'C', 'C','D'], 'Detail': ['d1', 'd2', 'd3', 'd4', 'd5', 'd6', 'd7', 'd8', 'd9','d0']} df = pd.DataFrame(data=d)  My goal is to use the substring of Food value without | and create a new dataframe where I can see which kitchens do produce similar foods. The way I define similarity is that substring should match with respect to Kitchen. df['Food'] = df['Food'].apply(str) df.insert(0,'subFood',df['Food'].str.split('|').str[0]) df.iloc[: , :2] subFood Food 0 P1 P1|0 1 P2 P2 2 P3 P3|45 3 P1 P1 4 P2 P2 5 P4 P4 6 P1 P1|1 7 P3 P3|7 8 P5 P5 9 P1 P1||23  To do so, I use merge function together with query. df.merge(df, on=['subFood', 'Kitchen'], suffixes=('_1', '_2')).query('City_1 != City_2') subFood Food_1 Kitchen City_1 Detail_1 Food_2 City_2 Detail_2 1 P1 P1|0 L1 A d1 P1|1 C d7 2 P1 P1|0 L1 A d1 P1||23 D d0 3 P1 P1|1 L1 C d7 P1|0 A d1 5 P1 P1|1 L1 C d7 P1||23 D d0 6 P1 P1||23 L1 D d0 P1|0 A d1 7 P1 P1||23 L1 D d0 P1|1 C d7 11 P3 P3|45 L9 A d3 P3|7 C d8 12 P3 P3|7 L9 C d8 P3|45 A d3  I got stuck here. My intention is to have a dataframe that should look similar to the dataframe shown below. I appreciate any help and / or hint. subFood Food_1 Food_2 Kitchen City Detail P1 P1|0 P1|0 L1 A d1 P1 P1|0 P1|1 L1 C d1 .... 

Answer: IIUC, you can split each row into two rows by combining the city names to a list and then using explode: merged = df.merge(df, on=["subFood","Kitchen"], suffixes=("_1","_2")).query("City_1 != City_2") merged["City"] = merged[["City_1","City_2"]].to_numpy().tolist() output = merged.drop(["City_1","City_2","Detail_2"],axis=1).explode("City").rename(columns={"Detail_1":"Detail"}) >>> output subFood Food_1 Kitchen Detail Food_2 City 1 P1 P1|0 L1 d1 P1|1 A 1 P1 P1|0 L1 d1 P1|1 C 2 P1 P1|0 L1 d1 P1||23 A 2 P1 P1|0 L1 d1 P1||23 D 3 P1 P1|1 L1 d7 P1|0 C 3 P1 P1|1 L1 d7 P1|0 A 5 P1 P1|1 L1 d7 P1||23 C 5 P1 P1|1 L1 d7 P1||23 D 6 P1 P1||23 L1 d0 P1|0 D 6 P1 P1||23 L1 d0 P1|0 A 7 P1 P1||23 L1 d0 P1|1 D 7 P1 P1||23 L1 d0 P1|1 C 11 P3 P3|45 L9 d3 P3|7 A 11 P3 P3|45 L9 d3 P3|7 C 12 P3 P3|7 L9 d8 P3|45 C 12 P3 P3|7 L9 d8 P3|45 A
Question: So I have what appears to be a perfectly acceptable loop to make parallel. But when I pass it to Numba parallel, it always gives incorrect results. All that happens in the loop is an input matrix has one element set to 0, matrix multiplication occurs and populates a new matrix, then the element that was set to 0 is set back to its original value. It would appear the array a is getting modified in each dispatch of Numba, so I tried copying a to another variable inside the loop, modifying the copy only, yet obtain the same incorrect results (not shown). Here is a minimal example. I just don't see what the issue is, or how to fix it: import numpy as np from scipy.stats import random_correlation import numba as nb def myfunc(a, corr): b = np.zeros(a.shape[0]) for i in range(b.shape[0]): temp = a[i] a[i] = 0 b[i] = a@corr@a.T a[i] = temp return b @nb.njit(parallel=True) def numbafunc(a, corr): b = np.zeros(a.shape[0]) for i in nb.prange(b.shape[0]): temp = a[i] a[i] = 0 b[i] = a@corr@a.T a[i] = temp return b if __name__ == '__main__': a = np.random.rand(10) corr = random_correlation.rvs(eigs=[2,2,1,1,1,1,0.5,0.5,0.5,0.5]) b_1 = myfunc(a, corr) b_2 = numbafunc(a, corr) # check if serial and Numba results match off the same inputs print(np.isclose(b_1,b_2)) # double check the original function returns the same results again.. b_1_check = myfunc(a, corr) print(np.isclose(b_1, b_1_check))  Returns all false values, or at least 9/10 are false... Can anyone pinpoint which part of the code is problematic for parallelization? It looks fine to me. Much appreciated!

Answer: There is a race condition in numbafunc. Indeed, a[i] = 0 modifies the array a shared between multiple threads reading/writing a for different i values. Storing the value in temp to restore it later only works in sequential, but not in parallel since threads can read a at any time. To solve this issue, each thread should operate on its own copy of a: @nb.njit(parallel=True) def numbafunc(a, corr): b = np.zeros(a.shape[0]) for i in nb.prange(b.shape[0]): c = a.copy() c[i] = 0.0 b[i] = c @ corr @ c.T return b
Question: I understand that a Dict parameter in a Python function is best set to a default of None. However, mypy seems to disagree:  def example(self, mydict: Dict[int, str] = None): return mydict.get(1)  This results in the mypy error: error: Incompatible default for argument "synonyms" (default has type "None", argument has type "Dict[Any, Any]") [assignment]  mypy on the other hand is fine with this:  def example(self, myDict: Dict[int, str] = {}):  But pylint complains: W0102: Dangerous default value {} as argument (dangerous-default-value)  According to this SO question (https://stackoverflow.com/a/26320917), the default should be None but that will not work with mypy. What is the option that will satisfy both pylint and mypy? Thanks. SOLUTION (based on comments): class Example: """Example""" def __init__(self): self.d: Optional[Dict[int, str]] = None def example(self, mydict: Optional[Dict[int, str]] = None): """example""" self.d = mydict  Part of the issue (not mentioned originally) I had was assigning back to a previously inited variable, this works.

Answer: I think the type should be "dict or None", so a Union of the two: def example(self, mydict: Union[Dict[int, str], None] = None): return mydict.get(1)
Question: I googled how to install the fbprophet package, but the top result is how to install prophet. What is the difference between the two packages? Are they the same?

Answer: It's by the same devs. Seems it was just a name change.  Prophet is on PyPI, so you can use pip to install it. From v0.6 onwards, Python 2 is no longer supported. As of v1.0, the package name on PyPI is "prophet"; prior to v1.0 it was "fbprophet".  https://pythonlang.dev/repo/facebook-prophet/