eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I was asked to find the total number of substring (case insensitive with/without punctuations) occurrences in a given string. Some examples: count_occurrences("Text with", "This is an example text with more than +100 lines") # Should return 1 count_occurrences("'example text'", "This is an 'example text' with more than +100 lines") # Should return 1 count_occurrences("more than", "This is an example 'text' with (more than) +100 lines") # Should return 1 count_occurrences("clock", "its 3o'clock in the morning") # Should return 0  I chose regex over .count() as I needed an exact match, and ended up with: def count_occurrences(word, text): pattern = f"(?<![a-z])((?<!')|(?<='')){word}(?![a-z])((?!')|(?=''))" return len(re.findall(pattern, text, re.IGNORECASE))  and I've got every matching count but my code took 0.10secs while expected time is 0.025secs. Am I missing something? is there any better (performance optimised) way to do this?

Answer: OK, I was struggling to make it work without regexes, as we all know that regexes are slow. Here is what I came up with: def count_occurrences(word, text): spaces = [' ', '\n', '(', '«', '\u201d', '\u201c', ':', "''", "__"] endings = spaces + ['?', '.', '!', ',', ')', '"', '»'] s = text.lower().split(word.lower()) l = len(s) return sum(( (i == 0 and (s[0] == '' or any(s[i].endswith(t) for t in spaces)) and (s[1] == '' or any(s[i+1].startswith(t) for t in endings))) or (i == l - 2 and any(s[i].endswith(t) for t in spaces) and (s[i+1] == '' or any(s[i+1].startswith(t) for t in endings))) or (i != 0 and i != l - 2 and any(s[i].endswith(t) for t in spaces) and any(s[i+1].startswith(t) for t in endings)) ) for i in range(l - 1))  The whole file runs in ideone: Ran 1 test in 0.025s OK  Which is what the question is asking for. The logic is pretty simple. Let's split the text by word, both lower cased. Now let's look at each couple of neighbours. If, for example index 0 finished with a valid delimiter, and index 1 starts with a valid delimiter, let's count it as an occurrence. Let's do that up to the last couple of the split. Since performance is important here, we have to be aware to the order of spaces and endings. We are basically looking for the first in the list to fulfil the condition. So it is important to locate the variables that are more common first. For example, If I declare: spaces = ['(', '«', '\u201d', '\u201c', ':', "''", "__", '\n', ' ']  instead of what I have in my solution, I get a run of 0.036 seconds. If for example I declare one array: spaces = [' ', '\n', '(', '«', '\u201d', '\u201c', ':', "''", "__", '?', '.', '!', ',', ')', '"', '»']  which has all delimiters and use only that, I get 0.053 seconds. Which is 60% more than my solution. It is probably possible that there is a better solution with declaring the delimiters in another order.
Question: I am trying to specify mypy type hints for the pytest native fixtures I am using in my test project e.g.: import pytest def pytest_configure(config): # Do something useful here  The config fixture returns a _pytest.config.Config object. If I try to model this naively: import pytest def pytest_configure(config: Config) -> None: # Do something useful here  I receive a mypy error: conftest.py:3: error: Name 'Config' is not defined [name-defined] I could do from _pytest.config import Config, but this doesn't seem to be a good way, because _pytest is private. Another option would be to ignore the type with # type: ignore. If this is the recommended way I would of course do this, but I wonder if there is a better option. I have the same issues in with any kind of pytest native fixtures I use, e.g. request which is used for parameterized fixtures. This would be a _pytest.fixtures.FixtureRequest.

Answer: Importing from _pytest.config Since pytest doesn't currently export Config (as of 6.2), the only way for typing is to use from _pytest.config import Config. This is how I also type config, as can be seen e.g. in this question of mine: from _pytest.config import Config def pytest_configure(config: Config) -> None: ...  You can track the typing progress in this pytest issue: #7469. Custom type stubs You can also introduce a small custom type stub that hides the reexport. It's questionable whether it will be useful here, only worth to mention for an alternative solution. If you create a file _typeshed/pytest.pyi with the following contents: from typing import Any from _pytest.config import Config as Config def __getattr__(name: str) -> Any: ... # incomplete  and make it accessible to mypy in mypy.ini: [mypy] mypy_path = _typeshed  Now you can import from pytest import Config at least in type checking mode - the runtime import will still fail. So the imports would look like from typing import Any, TYPE_CHECKING if TYPE_CHECKING: from pytest import Config else: Config = Any def pytest_configure(config: Config) -> None: pass  The only benefit of that solution is that the private import is now hidden; I'd still go with the private import though.
Question: I have an XML document and I want to get the line at which the tag extracted by BeautifulSoup or lxml is found. Is there a way to do that?

Answer: For BeautifulSoup this attribute is stored in the sourceline attribute of the Tag class, and is being populated in the parsers here and here. For lxml this is also possible through the sourceline attribute. Here is an example: #!/usr/bin/python3 from lxml import etree xml = ''' <a> <b> <c> </c> </b> <d> </d> </a> ''' root = etree.fromstring(xml) for e in root.iter(): print(e.tag, e.sourceline)  Output: a 2 b 3 c 4 d 7  If you want to look at the implementation of the sourceline method it's actually calling xmlGetLineNo which is a binding of xmlGetLineNo from libxml2 that is a wrapper for xmlGetLineNoInternal (where the actual logic for this lives inside libxml2). You can find the line number of the closing tag as well by checking how many line endings there are in the text representation of the subtree of that tag. xml.etree.ElementTree can also be extended to provide the line numbers where the elements have been found by the parser (the parser being xmlparser from the module xml.parsers.expat ).
Question: I was using the keyword built-in module to get a list of all the keywords of the current Python version. And this is what I did: >>> import keyword >>> print(keyword.kwlist) ['False', 'None', 'True', '__peg_parser__', 'and', 'as', 'assert', 'async', 'await', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']  And in the keyword.kwlist list there is __peg_parser__. So to see what it does, I type __peg_parser__ in a Python 3.9 interpreter on Windows (you'll get the same output on Mac OS and Linux as well), and this is what is get: >>> __peg_parser__ File "<stdin>", line 1 __peg_parser__ ^ SyntaxError: You found it!  So my question is, what is __peg_parser__ and why do I get SyntaxError: You found it!?

Answer: It was an easter egg related to the rollout of the new PEG parser. The easter egg, along with the old LL(1) parser, will be removed in 3.10.
Question: I try to solve an equation with solve from Sympy. But my approach doesn't work as desired. My equation : 0.00622765954483725 = (x * 24.39 * 0.921107170819325) / 143860432.178345. My code : from sympy import symbols, solve import numpy as np x = symbols('x') sol = solve((np.array([[x],[x]]) * np.array([[24.39],[293.6]]) * np.array([[0.921107170819325],[1]])) / np.array([[143860432.178345],[143860432.178345]]) - np.array([[0.00622765954483725],[0.0089267519953503]]))  I had success with a linear expression, but I have a DataFrame and I want to solve all data at the same time. from sympy import symbols, solve x = symbols('x') sol = solve((x * 24.39 * 0.921107170819325) / 143860432.178345 - 0.00622765954483725) 

Answer: Numpy doesn't understand about sympy's symbols, nor does sympy understand about numpy arrays. The only way to make them work together, is with sympy's lambdify which can convert a symbolic sympy expression to a numpy function. In your case, you first need to create a symbolic solution, lambdify it, and call it on your arrays: from sympy import symbols, solve, Eq, lambdify a, b, c, d = symbols('a b c d', real=True) x = symbols('x') sol = solve(Eq(x * a * b / c, d), x) # solve returns a list of solutions, in this case a list with just one element np_sol = lambdify((a, b, c, d), sol[0]) # convert the first solution to a (numpy) function # everything before is only sympy, everything from here is only numpy import numpy as np a_np = np.array([[24.39], [293.6]]) b_np = np.array([[0.921107170819325], [1]]) c_np = np.array([[143860432.178345], [143860432.178345]]) d_np = np.array([[0.00622765954483725], [0.0089267519953503]]) np_sol(a_np, b_np, c_np, d_np)  Result: array([[39879.], [ 4374.]])
Question: It's a great hassle when installing some packages in a VE and conda or pip downloads them again even when I already have it in my base environment. Since I have limited internet bandwidth and I'm assuming I'll work with many different VE's, it will take a lot of time to download basic packages such as OpenCV/Tensorflow.

Answer: By default, pip caches anything it downloads, and will used the cached version whenever possible. This cache is shared between your base environment and all virtual environments. So unless you pass the --no-cache-dir option, pip downloading a package means it has not previously downloaded a compatible version of that package. If you already have that package installed in your base environment or another virtual environment and it downloads it anyway, this probably means one or more of the following is true:  You installed your existing version with a method other than pip. There is a newer version available, and you didn't specify, for example, pip install pandas=1.1.5 (if that's the version you already have elsewhere). Pip will install the newest compatible version for your environment, unless you tell it otherwise. The VE you're installing to is a different Python version (e.g. created with Pyenv), and needs a different build.  I'm less familiar with the specifics of conda, and I can't seem to find anything in its online docs that focuses on the default caching behavior. However, a how-to for modifying the cache location seems to assume that the default behavior is similar to how pip works. Perhaps someone else with more Anaconda experience can chime in as well. So except for the caveats above, as long as you're installing a package with the same method you did last time, you shouldn't have to download anything. If you want to simplify the process of installing all the same packages (that were installed via pip) in a new VE that you already have in another environment, pip can automate that too. Run pip freeze > requirements.txt in the first environment, and copy the resulting file to your newly created VE. There, run pip install -r requirements.txt and pip will install all the packages that were installed (via pip) in the first environment. (Note that pip freeze records version numbers as well, so this won't install newer versions that may be available -- whether this is a good or bad thing depends on your needs.)
Question: I have the following DataFrame:     userId column_1 column_2 column_3     A 4.959 3.231 1.2356   B 0.632 0.963 2.4556   C 3.234 7.445 5.3435   D 1.454 0.343 2.2343     I would like to argsort w.r.t columns from the previous one:     userId first second third     A column_3 column_2 column_1   B column_1 column_2 column_3   C column_1 column_3 column_2   D column_2 column_1 column_3    

Answer: You can use np.argsort over axis 1. Then convert df.columns to numpy array using pd.Index.to_numpy and use numpy indexing. df = df.set_index('userId') # If userId is not index already. idx = df.values.argsort(axis=1) out = pd.DataFrame(df.columns.to_numpy()[idx], index=df.index) 0 1 2 userId A column_3 column_2 column_1 B column_1 column_2 column_3 C column_1 column_3 column_2 D column_2 column_1 column_3
Question: I can't find how to import AnalysisException in PySpark so I can catch it. For example: df = spark.createDataFrame([[1, 2], [1, 2]], ['A', 'A']) try: df.select('A') except AnalysisException as e: print(e)  Error message: NameError: name 'AnalysisException' is not defined 

Answer: You can import it here: from pyspark.sql.utils import AnalysisException  This is shown in the error traceback like Traceback (most recent call last): ... File "<string>", line 3, in raise_from pyspark.sql.utils.AnalysisException: cannot resolve ...
Question: I'm trying to use FTPSHook to send file through FTP TLS/SSL Explicit Encryption. Here's my code remote_filepath=pathfile local_filepath=pathfile2 hook = FTPSHook(ftp_conn_id='ftp_test') hook.store_file(remote_filepath, local_filepath)  and I'm getting this error when I run the DAG:  522 SSL/TLS required on the data channel  Does anyone ever done this before? How can I secure the connection with FTPSHook?

Answer: The ftplib (the underlying implementation of FTP(S) for the FTPSHook) does not encrypt the FTP data connection by default. To enable it, you have to call FTP_TLS.prot_p(). With FTPSHook API, you do it like this: hook = FTPSHook(ftp_conn_id='ftp_test') hook.get_conn().prot_p()
Question: I installed pywinauto using pip install pywinauto. OS: Windows 10 Python: 3.6.2 When I run python and try to import pywinauto, I get the error: Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from pywinauto.application import Application Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\__init__.py", line 89, in <module> from . import findwindows File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\findwindows.py", line 42, in <module> from . import controls File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\controls\__init__.py", line 36, in <module> from . import uiawrapper # register "uia" back-end (at the end of uiawrapper module) File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\controls\uiawrapper.py", line 47, in <module> from ..uia_defines import IUIA File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\uia_defines.py", line 181, in <module> pattern_ids = _build_pattern_ids_dic() File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\uia_defines.py", line 169, in _build_pattern_ids_dic if hasattr(IUIA().ui_automation_client, cls_name): File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\uia_defines.py", line 50, in __call__ cls._instances[cls] = super(_Singleton, cls).__call__(*args, **kwargs) File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\pywinauto\uia_defines.py", line 60, in __init__ self.UIA_dll = comtypes.client.GetModule('UIAutomationCore.dll') File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\comtypes\client\_generate.py", line 118, in GetModule mod = _CreateWrapper(tlib, pathname) File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\comtypes\client\_generate.py", line 183, in _CreateWrapper generate_module(tlib, ofi, pathname) File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\comtypes\tools\tlbparser.py", line 750, in generate_module gen.generate_code(list(items.values()), filename=pathname) File "C:\Users\John\AppData\Roaming\Python\Python36\site-packages\comtypes\tools\codegenerator.py", line 261, in generate_code tlib_mtime = os.stat(self.filename).st_mtime FileNotFoundError: [WinError 2] The system cannot find the file specified: 'UIAutomationCore.dll'  Any ideas how to fix this?

Answer: I have the same issue today and fixed it by pip install comtypes==1.1.7. It caused by comtypes library which release a new version 1.1.8 at Dec.26. Downgrade to previous version, it works well now.
Question: I have the following time series:  Date Value 0 2006-01-03 18 1 2006-01-04 12 2 2006-01-05 11 3 2006-01-06 10 4 2006-01-09 22 ... ... ... 3510 2019-12-23 47 3511 2019-12-24 46 3512 2019-12-26 35 3513 2019-12-27 35 3514 2019-12-30 28  I want to calculate the average values per month. So the pseudocode for each month is as follows:  Sum all the values for each day present in that month Divide by the number of days with data for that month.  The desired output would be something similar to:  Date Value 0 2006-01 17.45 1 2006-02 18.23 2 2006-04 16.79 3 2006-05 17.98 ... ... ... 166 2019-11 37.89 167 2019-12 36.34  I have tried this without success: data = data.set_index('Date') data.resample('M') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-28-435afe449f1f> in <module> 47 data = pd.DataFrame(dataList, columns=('Date', 'Value')) 48 data = data.set_index('Date') ---> 49 data.resample('M') 

Answer: We can convert your datetime column into a PeriodIndex on monthly frequency, then take the mean using GroupBy.mean: df.groupby(pd.PeriodIndex(df['Date'], freq="M"))['Value'].mean() Date 2006-01 14.6 2019-12 38.2 Freq: M, Name: Value, dtype: float64   df.groupby(pd.PeriodIndex(df['Date'], freq="M"))['Value'].mean().reset_index() Date Value 0 2006-01 14.6 1 2019-12 38.2  One caveat of this approach is that missing months are not shown. If that's important, use set_index and resample.mean in the same way.
Question: I have the following data frame df = pd.DataFrame([[1990,7,1000],[1990,8,2500],[1990,9,2500],[1990,9,1500],[1991,1,250],[1991,2,350],[1991,3,350],[1991,7,450]], columns = ['year','month','data1']) year month data1 1990 7 1000 1990 8 2500 1990 9 2500 1990 9 1500 1991 1 250 1991 2 350 1991 3 350 1991 7 450  I would like to filter the data such that it won't contain data with month/year 07/1990, 08/1990 and 01/1991. I can do for each combination month/year as follow: df = df.loc[(df.year != 1990) | (df.month != 7)]  But it is not efficient if there are many combinations month/year. Is there any more efficient way of doing this? Many thanks.

Answer: You could do: mask = ~df[['year', 'month']].apply(tuple, 1).isin([(1990, 7), (1990, 8), (1991, 1)]) print(df[mask])  Output  year month data1 2 1990 9 2500 3 1990 9 1500 5 1991 2 350 6 1991 3 350 7 1991 7 450
Question: I am trying to create a model which predicts results column below:  Date Open High Close Result 1/22/2010 25.95 31.29 30.89 0.176104 2/19/2010 23.98 24.22 23.60 -0.343760 3/19/2010 21.46 23.16 22.50 0.124994 4/23/2010 21.32 21.77 21.06 -0.765601 5/21/2010 55.41 55.85 49.06 0.302556  The code I am using is: import pandas from sklearn.tree import DecisionTreeClassifier dataset = pandas.read_csv('data.csv') X = dataset.drop(columns=['Date','Result']) y = dataset.drop(columns=['Date', 'Open', 'High', 'Close']) model = DecisionTreeClassifier() model.fit(X, y)  But I am getting an error: ValueError: Unknown label type: 'continuous'  Suggestion for using other algorithms are also welcome.

Answer: In ML, it's important as a first step to consider the nature of your problem. Is it a regression or classification problem? Do you have target data (supervised learning) or is this a problem where you don't have a target and want to learn more about your data's inherent structure (such as unsupervised learning). Then, consider what steps you need to take in your pipeline to prepare your data (preprocessing). In this case, you are passing floats (floating point numbers) to a Classifier (DecisionTreeClassifier). The problem with this is that a classifier generally separates distinct classes, and so this classifier expects a string or an integer type to distinguish different classes from each other (this is known as the "target"). You can read more about this in an introduction to classifiers. The problem you seek to solve is to determine a continuous numerical output, Result. This is known as a regression problem, and so you need to use a Regression algorithm (such as the DecisionTreeRegressor). You can try other regression algorithms out once you have this simple one working, and this is a good place to start as it is a fairly straight forward one to understand, it is fairly transparent, it is fast, and easily implemented - so decision trees were a great choice of starting point! As a further note, it is important to consider preprocessing your data. You have done some of this simply by separating your target from your input data: X = dataset.drop(columns=['Date','Result']) y = dataset.drop(columns=['Date', 'Open', 'High', 'Close'])  However, you may wish to look into preprocessing further, particularly standardisation of your data. This is often a required step for whichever ML algorithm you implement to be able to interpret your data. There's a saying that goes: "Garbage in, garbage out". Part of preprocessing sometimes requires you to change the data type of a given column. The error posted in your question, at face value, leads one to think that the issue on hand is that you need to change data types. But, as explained, in the case of your problem, it wouldn't help to do that, given that you seek to use regression to determine a continuous output.
Question: I have a column in my dataframe being the customer ids which contains no repetitions. The id series starts at integer 1, and ends at 4003. As the following output shows, there are 4 id numbers being skipped. I would like some help in finding out what they are. Thanks in advance! df['customer_id'].describe() Out[150]: count 3999 unique 3999 top 4003 freq 1 Name: customer_id, dtype: int64 

Answer: Assuming the dtype is int (which appears to be the case), it looks like we can use setdiff1d here from numpy: c_id = df['customer_id'] missing_ids = np.setdiff1d(np.arange(c_id.min(), c_id.max()+1), c_id)