eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My question is similar to this. I also use pickle to save & load model. I meet the below error during pickle.load( ) from sklearn.preprocessing import StandardScaler # SAVE scaler = StandardScaler().fit(X_train) X_trainScale = scaler.transform(X_train) pickle.dump(scaler, open('scaler.scl','wb')) # ================= # LOAD sclr = pickle.load(open('scaler.scl','rb')) # => ModuleNotFoundError: No module named 'sklearn.preprocessing._data' X_testScale = sclr.transform(X_test)  ModuleNotFoundError: No module named 'sklearn.preprocessing._data' It looks like a sklearn version issue. My sklearn version is 0.20.3, Python version is 3.7.3. But I am using Python in an Anaconda .zip file. Is it possible to solve this without updating the version of sklearn?

Answer: I had exactly the same error message with StandardScaler using Anaconda.  Fixed it by running: conda update --all  I think the issue was caused by running the pickle dump for creating the scaler file on a machine with a newer version of scikit-learn, and then trying to run pickle load on machine with an older version of scikit-learn. (It gave the error when running pickle load on the machine with the older version of scikit-learn but no error when running pickle load on the machine with the newer version of scikit-learn. Both windows machines). Perhaps this is due to more recent versions using a different naming convention for functions regarding underscores (as mentioned above)? Anaconda would not let me update the scikit-learn library on it's own, because it claimed it required the older version (for some reason I could not understand). Perhaps another library was using it? So I had to fix it by updating all the libraries at the same time, which worked.
Question: What is the correct way to type an "interface" in python 3? In the following sample: class One(object): def foo(self) -> int: return 42 class Two(object): def foo(self) -> int: return 142 def factory(a: str): if a == "one": return One() return Two()  what would be the correct way to type the return value of the factory function? It should be something like "A type with a single method named foo that accepts no arguments and returns an integer". But not sure I can find how to do that. UPD: this question is exclusively dedicated to typing.

Answer: You could use a typing.Union but, it sounds like you really want structural typing not nominal. Python supports this using typing.Protocol, which is a supported part of the python type-hinting system, so mypy will understand it, for example: import typing class Fooable(typing.Protocol): def foo(self) -> int: ... class One(object): def foo(self) -> int: return 42 class Two(object): def foo(self) -> int: return 142 def factory(a: str) -> Fooable: if a == "one": return One() return Two() x = factory('one') x.foo()  Note, structural typing fits well with Python's duck-typing ethos. Python's typing system supports both structural and nominal forms.
Question: I am so confused with different indexing methods using iloc in pandas.  Let say I am trying to convert a 1-d Dataframe to a 2-d Dataframe. First I have the following 1-d Dataframe a_array = [1,2,3,4,5,6,7,8] a_df = pd.DataFrame(a_array).T  And I am going to convert that into a 2-d Dataframe with the size of 2x4. I start by preseting the 2-d Dataframe as follow: b_df = pd.DataFrame(columns=range(4),index=range(2))  Then I use for-loop to help me converting a_df (1-d) to b_df (2-d) with the following code for i in range(2): b_df.iloc[i,:] = a_df.iloc[0,i*4:(i+1)*4]  It only gives me the following results  0 1 2 3 0 1 2 3 4 1 NaN NaN NaN NaN  But when I changed b_df.iloc[i,:] to b_df.iloc[i][:]. The result is correct like the following, which is what I want  0 1 2 3 0 1 2 3 4 1 5 6 7 8  Could anyone explain to me what the difference between .iloc[i,:] and .iloc[i][:] is, and why .iloc[i][:] worked in my example above but not .iloc[i,:] 

Answer: There is a very, very big difference between series.iloc[:] and series[:], when assigning back. (i)loc always checks to make sure whatever you're assigning from matches the index of the assignee. Meanwhile, the [:] syntax assigns to the underlying NumPy array, bypassing index alignment. s = pd.Series(index=[0, 1, 2, 3], dtype='float') s 0 NaN 1 NaN 2 NaN 3 NaN dtype: float64 # Let's get a reference to the underlying array with `copy=False` arr = s.to_numpy(copy=False) arr # array([nan, nan, nan, nan]) # Reassign using slicing syntax s[:] = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd']) s 0 1 1 2 2 3 3 4 dtype: int64 arr # array([1., 2., 3., 4.]) # underlying array has changed # Now, reassign again with `iloc` s.iloc[:] = pd.Series([5, 6, 7, 8], index=[3, 4, 5, 6]) s 0 NaN 1 NaN 2 NaN 3 5.0 dtype: float64 arr # array([1., 2., 3., 4.]) # `iloc` created a new array for the series # during reassignment leaving this unchanged s.to_numpy(copy=False) # the new underlying array, for reference # array([nan, nan, nan, 5.])   Now that you understand the difference, let's look at what happens in your code. Just print out the RHS of your loops to see what you are assigning: for i in range(2): print(a_df.iloc[0, i*4:(i+1)*4]) # output - first row 0 1 1 2 2 3 3 4 Name: 0, dtype: int64 # second row. Notice the index is different 4 5 5 6 6 7 7 8 Name: 0, dtype: int64  When assigning to b_df.iloc[i, :] in the second iteration, the indexes are different so nothing is assigned and you only see NaNs. However, changing b_df.iloc[i, :] to b_df.iloc[i][:] will mean you assign to the underlying NumPy array, so indexing alignment is bypassed. This operation is better expressed as  for i in range(2): b_df.iloc[i, :] = a_df.iloc[0, i*4:(i+1)*4].to_numpy() b_df 0 1 2 3 0 1 2 3 4 1 5 6 7 8  It's also worth mentioning this is a form of chained assignment, which is not a good thing, and also makes your code harder to read and understand.
Question: im very new to python, trying to create reusable code. when i try to call the class Login and function login_user in test_main.py by passing all the arguments that were used under Login class, im getting an error as InvalidArgumentException: Message: invalid argument: 'using' must be a string. test_main.py file which runs on pytest. Locators_test is the class of test_Locators.py file where i have all my xpaths test_Locators.py class Locators_test(): loginlink_xpath = "//a[@id='login-link']" login_email = "xxxxx" login_password = "xxxxx" loginemail_id = "dnn_ctr1179_Login_txtEmail" loginpassword_id = "dnn_ctr1179_Login_txtPassword" clicklogin_id = "dnn_ctr1179_Login_btnLogin"  test_login.py from Smoketest.locatorfile.test_Locators import Locators_test class Login(): def __init__(self,driver): self.driver = driver def login_user(self,driver): try: loginButton = self.driver.find_element((By.XPATH, Locators_test.loginlink_xpath)) while loginButton.click() is True: break time.sleep(3) self.driver.execute_script("window.scrollBy(0,300);") EmailField = self.driver.find_element((By.ID, Locators_test.loginemail_id)) EmailField.send_keys(Locators_test.login_email) PasswordField = self.driver.find_element((By.ID, Locators_test.loginpassword_id)) PasswordField.send_keys(Locators_test.login_password) ClickLogin = self.driver.find_element((By.ID, Locators_test.clicklogin_id)) while ClickLogin.click() is True: break time.sleep(5) userName = self.driver.find_element((By.XPATH, Locators_test.username_xpath)) print("Logged in as", userName.text) except StaleElementReferenceException or ElementClickInterceptedException or TimeoutException as ex: print(ex.message)  test_main.py  def test_setup(): driver = webdriver.Chrome(executable_path= Locators_test.browser_path) driver.maximize_window() driver.delete_all_cookies() driver.get(homePage) driver.implicitly_wait(5) yield print("test complete") def test_login(test_setup): from Smoketest.pages.test_login import Login lo = Login(driver) lo.login_user(((Locators_test.loginlink_xpath,Locators_test.loginemail_id,Locators_test.login_email,Locators_test.loginpassword_id,Locators_test.login_password,Locators_test.clicklogin_id,Locators_test.username_xpath)))  indentations are all fine

Answer: I fixed it myself by removing the extra pair of parenthesis from the line loginButton = self.driver.find_element((By.XPATH, Locators_test.loginlink_xpath)) Right way is loginButton = self.driver.find_element(By.XPATH, Locators_test.loginlink_xpath) ps: this applies to all the lines.
Question: My model is trained on digit images (MNIST dataset). I am trying to print the output of the second layer of my network - an array of 128 numbers. After reading a lot of examples - for instance this, and this, or this. I did not manage to do this on my own network. Neither of the solutions work of my own algorithm. Link to Colab: https://colab.research.google.com/drive/1MLbpWJmq8JZB4_zKongaHP2o3M1FpvAv?fbclid=IwAR20xRz2i6sFS-Nm6Xwfk5hztdXOuxY4tZaDRXxAx3b986HToa9-IaTgASU I received a lot of different error messages. I tried to handle each of them, but couldn't figure it on my own. What am I missing? How to output the Second layer? If my Shape is (28,28) - what should be the type & value of input_shape?  Failed trials & Errors for example: (1)  for layer in model.layers: get_2nd_layer_output = K.function([model.layers[0].input],[model.layers[2].output]) layer_output = get_2nd_layer_output(layer)[0] print('\nlayer output: get_2nd_layer_output=, layer=', layer, '\nlayer output: get_2nd_layer_output=', get_2nd_layer_output)   TypeError: inputs should be a list or tuple.  (2) input_shape=(28, 28) inp = model.input # input placeholder outputs = [layer.output for layer in model.layers] # all layer outputs functor = K.function([inp, K.learning_phase()], outputs ) # evaluation function # Testing test = np.random.random(input_shape)[np.newaxis,...] layer_outs = functor([test, 0.]) print('layer_outs',layer_outs)   tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable dense_1/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/dense_1/bias) [[{{node dense_1/BiasAdd/ReadVariableOp}}]] 

Answer: Looks like you are mixing old keras (before tensorflow 2.0: import keras) and new keras (from tensorflow import keras).  Try not to use old keras alongside tensorflow>=2.0 (and not to refer to the old documentation as in your first link), as it is easily confused with the new one (although nothing strictly illogical):  from tensorflow import keras from keras.models import Model print(Model.__module__) #outputs 'keras.engine.training' from tensorflow.keras.models import Model print(Model.__module__) #outputs 'tensorflow.python.keras.engine.training'  Behaviour will be highly unstable mixing those two libraries. Once this is done, using an answer from what you tried, m being your model, and my_input_shape being the shape of your models input ie the shape of one picture (here (28, 28) or (1, 28, 28) if you have batches): from tensorflow import keras as K my_input_data = np.random.rand(*my_input_shape) new_temp_model = K.Model(m.input, m.layers[3].output) #replace 3 with index of desired layer output_of_3rd_layer = new_temp_model.predict(my_input_data) #this is what you want  If you have one image img you can directly write new_temp_model.predict(img)
Question: I am trying to run Flake8 for my python code however I'm noticing it's not giving me any of the PyDocStyle errors on a simple class with missing docstrings or warning about my class name cars which should be Cars according to PEP8 style guide Example code file (cars.py) class cars: def __init__(self, some_value): self.some_value = some_value  when I run flake8 cars.py I get the following output: cars.py:3:37: W292 no newline at end of file  I'm trying to configure Flake8 to run some common style checks but not finding any help in the Flake8 documentation about how to enable PyDocStyle error codes. For comparison, I ran the same file against Pylint and here is the output ************* Module code.cars cars.py:3:0: C0304: Final newline missing (missing-final-newline) cars.py:1:0: C0114: Missing module docstring (missing-module-docstring) cars.py:1:0: C0103: Class name "cars" doesnt conform to PascalCase naming style (invalid-name) cars.py:1:0: C0115: Missing class docstring (missing-class-docstring) cars.py:1:0: R0903: Too few public methods (0/2) (too-few-public-methods) ------------------------------------ Your code has been rated at -6.67/10  I'm using python 3.7.6, flake8 3.7.9 (mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.6 on Linux

Answer: So what I found out was that by default Flake8 wraps pycodestyle: 2.5.0 by default which from the documentation says:  Among other things, these features are currently not in the scope of the pycodestyle library:  naming conventions: this kind of feature is supported through plugins. Install flake8 and the pep8-naming extension to use this feature. docstring conventions: they are not in the scope of this library; see the pydocstyle project.  automatic fixing: see the section PEP8 Fixers in the related tools page.   So I installed pep8-naming, as well as flake8-docstrings and after running flake8 --version I got the below which shows it is now using the installed plugins: 3.7.9 (flake8-docstrings: 1.5.0, pydocstyle: 5.0.2, mccabe: 0.6.1, naming: 0.8.2, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.6 on Darwin  I reran my check flake8 cars.py and I got the below output: cars.py:1:1: D100 Missing docstring in public module cars.py:2:1: D101 Missing docstring in public class cars.py:2:8: N801 class name 'cars' should use CapWords convention cars.py:3:1: D107 Missing docstring in __init__  At a first impression - after checking out the git repos for flake8 and the additional plugins I had to install I am slightly skeptical on Flake8. The reason being is as at the time of writing Pylint seems to come packaged with the behavior I need as well as being around longer which benefits from stability and more contributors. Contrast to flake8 which is newer, and to achieve the desired behavior it is necessary to install 3rd party plugins/libraries that might be abandoned in a year or two or break when flake8 upgrades. Either scenario is problematic with the latter being troublesome if you're not careful with mentioning specific version/builds in CI pipelines.
Question: I read a file into a pandas dataframe with dates that vary in their format:  either the American: YYYY-MM-DD or the European: DD.MM.YYYY  They come as a string. I would like to format them all as a date object so pandas.Series.dt can work with them and ideally have them in the second format (DD.MM.YYYY). pandas.Series.dt gets confuesed with the two different spellings in one column.

Answer: Use to_datetime with both formats separately, so get missing values if format not match, so for new column use Series.fillna: df = pd.DataFrame({'date': ['2000-01-12', '2015-01-23', '20.12.2015', '31.12.2009']}) print (df) date 0 2000-01-12 1 2015-01-23 2 20.12.2015 3 31.12.2009 date1 = pd.to_datetime(df['date'], errors='coerce', format='%Y-%m-%d') date2 = pd.to_datetime(df['date'], errors='coerce', format='%d.%m.%Y') df['date'] = date1.fillna(date2) print (df) date 0 2000-01-12 1 2015-01-23 2 2015-12-20 3 2009-12-31   and ideally have them in the second format  Format of datetimes in python/pandas is by default YYYY-MM-DD, if need custom one it is possible, but values are converted to strings, so datetimelike functions failed: df['date'] = df['date'].dt.strftime('%d.%m.%Y') print (df) date 0 12.01.2000 1 23.01.2015 2 20.12.2015 3 31.12.2009 print (type(df.loc[0, 'date'])) <class 'str'>
Question: How can I replace the first occurrence of a character in every word? Say I have this string: hello @jon i am @@here or @@@there and want some@thing in '@here" # ^ ^^ ^^^ ^ ^  And I want to remove the first @ on every word, so that I end up having a final string like this: hello jon i am @here or @@there and want something in 'here # ^ ^ ^^ ^ ^  Just for clarification, "@" characters always appear together in every word, but can be in the beginning of the word or between other characters. I managed to remove the "@" character if it occurs just once by using a variation of the regex I found in Delete substring when it occurs once, but not when twice in a row in python, which uses a negative lookahead and negative lookbehind: @(?!@)(?<!@@)  See the output: >>> s = "hello @jon i am @@here or @@@there and want some@thing in '@here" >>> re.sub(r'@(?!@)(?<!@@)', '', s) "hello jon i am @@here or @@@there and want something in 'here"  So the next step is to replace the "@" when it occurs more than once. This is easy by doing s.replace('@@', '@') to remove the "@" from wherever it occurs again. However, I wonder: is there a way to do this replacement in one shot?

Answer: I would do a regex replacement on the following pattern: @(@*)  And then just replace with the first capture group, which is all continous @ symbols, minus one. This should capture every @ occurring at the start of each word, be that word at the beginning, middle, or end of the string. inp = "hello @jon i am @@here or @@@there and want some@thing in '@here" out = re.sub(r"@(@*)", '\\1', inp) print(out)  This prints: hello jon i am @here or @@there and want something in 'here
Question: I am trying to do autocorrelation using Julia and compare it to Python's result. How come they give different results? Julia code using StatsBase t = range(0, stop=10, length=10) test_data = sin.(exp.(t.^2)) acf = StatsBase.autocor(test_data)  gives 10-element Array{Float64,1}: 1.0 0.13254954979179642 -0.2030283419321465 0.00029587850872956104 -0.06629381497277881 0.031309038331589614 -0.16633393452504994 -0.08482388975165675 0.0006905628640697538 -0.1443650483145533  Python code from statsmodels.tsa.stattools import acf import numpy as np t = np.linspace(0,10,10) test_data = np.sin(np.exp(t**2)) acf_result = acf(test_data)  gives array([ 1. , 0.14589844, -0.10412699, 0.07817509, -0.12916543, -0.03469143, -0.129255 , -0.15982435, -0.02067688, -0.14633346]) 

Answer: This is because your test_data is different: Python: array([ 0.84147098, -0.29102733, 0.96323736, 0.75441021, -0.37291918, 0.85600145, 0.89676529, -0.34006519, -0.75811102, -0.99910501])  Julia: [0.8414709848078965, -0.2910273263243299, 0.963237364649543, 0.7544102058854344, -0.3729191776326039, 0.8560014512776061, 0.9841238290665676, 0.1665709194875013, -0.7581110212957692, -0.9991050130774393]  This happens because you are taking sin of enormous numbers. For example, with the last number in t being 10, exp(10^2) is ~2.7*10^43. At this scale, floating point inaccuracies are about 3*10^9. So if even the least significant bit is different for Python and Julia, the sin value will be way off. In fact, we can inspect the underlying binary values of the initial array t. For example, they differ in the third last value: Julia: julia> reinterpret(Int, range(0, stop=10, length=10)[end-2]) 4620443017702830535  Python: >>> import struct >>> s = struct.pack('>d', np.linspace(0,10,10)[-3]) >>> struct.unpack('>q', s)[0] 4620443017702830536  We can indeed see that they disagree by exactly one machine epsilon. And if we use Julia take sin of the value obtained by Python: julia> sin(exp(reinterpret(Float64, 4620443017702830536)^2)) -0.3400651855865199  We get the same value Python does.
Question: Trying to import a code from python2 to python 3 and this problem happens  <ipython-input-53-e9f33b00348a> in aesEncrypt(text, secKey) 43 def aesEncrypt(text, secKey): 44 pad = 16 - len(text) % 16 ---> 45 text = text.encode("utf-8") + (pad * chr(pad)).encode("utf-8") 46 encryptor = AES.new(secKey, 2, '0102030405060708') 47 ciphertext = encryptor.encrypt(text)   AttributeError:'bytes' object has no attribute 'encode'  If I remove .encode("utf-8") the error is "can't concat str to bytes". Apparently pad*chr(pad) seems to be a byte string. It cannot use encode()  <ipython-input-65-9e84e1f3dd26> in aesEncrypt(text, secKey) 43 def aesEncrypt(text, secKey): 44 pad = 16 - len(text) % 16 ---> 45 text = text.encode("utf-8") + (pad * chr(pad)) 46 encryptor = AES.new(secKey, 2, '0102030405060708') 47 ciphertext = encryptor.encrypt(text)   TypeError: can't concat str to bytes  However, the weird thing is that if i just try the part along. encode() works fine. text = { 'username': '', 'password': '', 'rememberLogin': 'true' } text=json.dumps(text) print(text) pad = 16 - len(text) % 16 print(type(text)) text = text + pad * chr(pad) print(type(pad * chr(pad))) print(type(text)) text = text.encode("utf-8") + (pad * chr(pad)).encode("utf-8") print(type(text)) {"username": "", "password": "", "rememberLogin": "true"} <class 'str'> <class 'str'> <class 'str'> <class 'bytes'> 

Answer: If you don't know if a stringlike object is a Python 2 string (bytes) or Python 3 string (unicode). You could have a generic converter. Python3 shell: >>> def to_bytes(s): ... if type(s) is bytes: ... return s ... elif type(s) is str or (sys.version_info[0] < 3 and type(s) is unicode): ... return codecs.encode(s, 'utf-8') ... else: ... raise TypeError("Expected bytes or string, but got %s." % type(s)) ... >>> to_bytes("hello") b'hello' >>> to_bytes("hello".encode('utf-8')) b'hello'  On Python 2 both these expressions evaluate to True: type("hello") == bytes and type("hello") == str. And type(u"hello") == str evaluates to False, while type(u"hello") == unicode is True. On Python 3 type("hello") == bytes is False, and type("hello") == str is True. And type("hello") == unicode raises a NameError exception since unicode isn't defined on 3. Python 2 shell: >>> to_bytes(u"hello") 'hello' >>> to_bytes("hello") 'hello'
Question: I am trying to wrap my head around how to write a context manager that deals with writing some logs while handling any exceptions. The problem I am trying to solve is to make code like this: try: # code that can raise exception here except Exception as e: print('failed', e) print('all good')  This is a repeated pattern I have in the code and I think it's best handled with a context manager like: with my_ctx_manager(success_msg='all good', failed_msg='failed): # code that can raise exception here  this looks much better, but I don't know how to write the actual context manager to deal with any exceptions that could rise inside the context. @contextlib.contextmanager def my_ctx_manager(success_msg, failed_msg): try: # if no exception then print(success_msg) # How do I catch any exception here except Exception: print(failed_msg) # I need the exception to propagate as well raise  I guess my question is more of the type: How do I make sure that the context manager correctly catches, logs and re-raise any exception for the code that is wrapping ?

Answer: The way the @contextmanager decorator works, you should write yield once within your context manager function, so that the with block will be executed while the yield statement pauses your function's execution. That means if the with block throws an exception, you can catch it by wrapping yield in a try/except block: from contextlib import contextmanager @contextmanager def example(): print('entered the context manager') managed_resource = 'some resource' try: yield managed_resource except Exception as e: print('caught:', e) # any cleanup that should only be done on failure raise else: # any cleanup that should only be done on success print('no exception was thrown') finally: # any cleanup that should always be done print('exited the context manager') with example() as resource: print('resource:', resource) raise ValueError('some error message')  Output: entered the context manager resource: some resource caught: some error message exited the context manager Traceback (most recent call last): File "<stdin>", line 3, in <module> ValueError: some error message  If you want to catch everything (not just Exception), then you can write a bare except: block and use sys.exc_info() to get the exception information.