eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am currently writing a python program which uses a seleniumbase web bot with CDP mode activated: with SB(uc=True, test=True, xvfb=True, incognito=True, agent=<user_agent>, headless=True) as sb: temp_email_gen_url = "https://temp-mail.org/en" sb.activate_cdp_mode(temp_email_gen_url) ...  I need to be able to create new tab and switch between the new and original tab. I have read the CDP docs but have not seen a solution to this, does anybody know how this can be done?

Answer: For better or worse there isn't an "open tab" feature in CDP mode. The main developer of seleniumbase suggests using a separate driver in CDP mode for each tab as follows, equivalent to using "open in new window" on every link: from seleniumbase import SB # opens all links on the target page with a second driver with SB(uc=True, test=True) as sb: temp_email_gen_url = "https://temp-mail.org/en" sb.driver.uc_open_with_reconnect(temp_email_gen_url) links = sb.get_unique_links() for link in links: driver2 = sb.get_new_driver(undetectable=True) driver2.uc_open_with_reconnect(link) print(driver2.title) sb.quit_extra_driver()  You may want to consider reusing the second driver for each link instead of creating and destroying a driver for each link. It would be faster and more efficient, but it's possible that the site could use cookies and session storage to detect a suspicious number of page accesses coming from the same browser session. To elaborate on a question in the comments: there is indeed a way in non-CDP mode to open tabs but I don't recommend it. Connecting the WebDriver leaves traces that bot detection scripts can find, both obvious (in years past hard-coded variable names were added to the JS environment) and subtle such as exploiting obscure behavior around how stack traces and logging commands are buffered and normally run lazily, but not if WebDriver is connected. seleniumbase's UC mode was an attempt at addressing this by using a WebDriver most of the time, but disconnecting for a while just before doing something that can result in detection then waiting until the danger is assumed to have passed before reconnecting. It worked for a while but hosting platforms have adapted. CDP mode is a relatively new entrant in this cat-and-mouse game that is much harder to detect. The growing counter to CDP mode is to track requests and UI interactions such as mouse movements and clicks over time and deploy models like recaptcha v3 that predict the probability a browser is a bot. The counter to that will be increased reliance on pyautogui and similar tools to simulate human interaction with the UI.
Question: I have set IPYTHONDIR=.ipython, and created a startup file at .ipython/profile_default/startup/01_hello.py. Now, when I run ipython, it executes the contents of that file as if they had been entered into the IPython shell. I can run sync code this way: # contents of 01_hello.py print( "hello!" )  $ ipython Python 3.12.0 (main, Nov 12 2023, 10:40:37) [GCC 11.4.0] Type 'copyright', 'credits' or 'license' for more information IPython 8.31.0 -- An enhanced Interactive Python. Type '?' for help. hello In [1]:  I can also run async code directly in the shell: # contents of 01_hello.py print( "hello!" ) async def foo(): print( "foo" )  $ ipython Python 3.12.0 (main, Nov 12 2023, 10:40:37) [GCC 11.4.0] Type 'copyright', 'credits' or 'license' for more information IPython 8.31.0 -- An enhanced Interactive Python. Type '?' for help. hello In [1]: await foo() foo In [2]:  However, I cannot run async code in the startup file, even though it's supposed to be as if that code was entered into the shell: # contents of 01_hello.py print( "hello!" ) async def foo(): print( "foo" ) await foo()  $ ipython Python 3.12.0 (main, Nov 12 2023, 10:40:37) [GCC 11.4.0] Type 'copyright', 'credits' or 'license' for more information IPython 8.31.0 -- An enhanced Interactive Python. Type '?' for help. [TerminalIPythonApp] WARNING | Unknown error in handling startup files: File ~/proj/.ipython/profile_default/startup/01_imports.py:5 await foo() ^ SyntaxError: 'await' outside function  Question: Why doesn't this work, and is there a way to run async code in the startup file without explicitly starting a new event loop just for that? (asyncio.run()) Doing that wouldn't make sense, since that event loop would have to close by the end of the file, which makes it impossible to do any initialization work that involves context vars (which is where Tortoise-ORM stores its connections), which defeats the purpose. Or stated differently: How can I access the event loop that IPython starts for the benefit of the interactive shell?

Answer: From version 8, ipython uses a function called get_asyncio_loop to get access to the event loop that it runs async cells on. You can use this event loop during your startup script to run any tasks you want on the same event loop that async cells will run on. NB. This is only uses for the asyncio package in Python's standard library and not any other async libraries (such as trio). from IPython.core.async_helpers import get_asyncio_loop as _get_asyncio_loop async def foo(): print("foo") _get_asyncio_loop().run_until_complete(foo())  Caveat The event loop that ipython uses DOES NOT run in the background. What this means is that unless you are running an async cell, no tasks that you have started will be running. ie. None of your Tortoise ORM connections will be serviced, which may cause them to break. As such, you may need to run your Tortoise ORM in a separate event loop anyway, and write some glue for passing data back and forth between the two event loops.
Question: I am running a Flask application on Docker with uwsgi. I have been running it for years now, but we need to add https to it. I know I can use an HAProxy and do ssl offloading, but in our current setup we cant do it this way, at least not right now. We need to do the SSL directly on the application. I have tried multiple options and I keep getting "The -s/--socket option is missing and stdin is not a socket." Not sure what else to try. The server is uWSGI==2.0.26. Please help. below is my uwsgi.ini file. [uwsgi] module = wsgi:app master = true processes = 5 enable-threads = true single-interpreter = true buffer-size = 32768 # protocol = http # socket = 0.0.0.0:5000 # protocol = https shared-socket = 0.0.0.0:5000 https = 0,/app/ssl/app_cert.crt,/app/ssl/app_cert.key stdout_logfile = /dev/stdout stdout_logfile_maxbytes = 0 stderr_logfile = /dev/stdout stderr_logfile_maxbytes = 0 chmod-socket = 660 vacuum = true die-on-term = true py-autoreload = 1 

Answer: You can use the following example to run your flask application with uwsgi and docker. I will provide a minimal example and you can use it to expand to your needs. The uwsgi conf were extracted from docs. uwsgi.ini [uwsgi] shared-socket = 0.0.0.0:443 https = =0,ssl/server.crt,ssl/server.key master = true module = app:app uid = uwsgi gid = uwsgi  app.py from flask import Flask app = Flask(__name__) @app.route("/") def hello(): return "Hello, World!" if __name__ == "__main__": app.run()  requirements.txt Flask uWSGI==2.0.26  Dockerfile FROM python:3.10-slim RUN apt-get update && apt-get install -y \ build-essential \ gcc \ libssl-dev \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* RUN groupadd -r uwsgi && useradd -r -g uwsgi -m uwsgi WORKDIR /app COPY requirements.txt /app/ RUN pip install --no-cache-dir -r requirements.txt COPY . /app/ COPY ssl/ /app/ssl/ RUN chown -R uwsgi:uwsgi /app EXPOSE 443 USER uwsgi CMD ["uwsgi", "--ini", "uwsgi.ini"]  Create an ssl directory and generate a self-signed cert. mkdir ssl openssl req -x509 \ -newkey rsa:2048 \ -keyout ssl/server.key \ -out ssl/server.crt \ -days 365 -nodes -subj "/CN=localhost"  Now you should have this folder structure: . ├── app.py ├── Dockerfile ├── requirements.txt ├── ssl │ ├── server.crt │ └── server.key └── uwsgi.ini  Now build and run: docker build -t flask-uwsgi-example . docker run --rm --name flask -p 443:443 flask-uwsgi-example  And test with curl: $ curl -k https://localhost:443 Hello, World!
Question: I'm using a quadratic function to fit my data. I have good R2 score but huge uncertainty in my fitting parameters Here is the graph and the results:  R2 score: 0.9698143924536671 uncertainty in a, b, and y0: 116.93787913, 10647.11867972, 116.93787935 How should I intepret this result? Here is how I defined the quadratic function: def my_quad(x, a, b, y0): return a*(1-x**2/(2*b**2))+ y0  Here's how I calculated the uncertainty for the parameters and R2 score: popt, pcov = curve_fit(my_quad, x_data,y_data, bounds=([0, 0, -np.inf], [np.inf, np.inf, np.inf])) a, b, y= popt err = np.sqrt(np.diag(pcov)) y_pred = my_quad(x_data, *popt) r2 = r2_score(y_data, y_pred)) 

Answer: Your model is over-parametrized. You can tell when you expand the polynomial: a * (1 - x**2 / (2*b**2)) + y0 -> a - x**2 * a / (2*b**2) + y0 -> y0+a - x**2 * a / (2*b**2)  The are only two independent parameters, y0 + a and a / (2*b**2). You will be able to fit just as well with any two of your original parameters, and then the uncertainty will be reduced significantly. For example: import numpy as np import matplotlib.pyplot as plt from scipy import stats from scipy.optimize import curve_fit # generate data rng = np.random.default_rng(23457834572346) x = np.linspace(-1, 1, 30) noise = 0.05 * rng.standard_normal(size=x.shape) y = -2*x**2 + 1 + noise # over-parameterized fit def my_quad(x, a, b, y0): return a*(1-x**2/(2*b**2))+ y0 bounds=([0, 0, -np.inf], [np.inf, np.inf, np.inf]) popt, pcov = curve_fit(my_quad, x, y, bounds=bounds) err = np.sqrt(np.diag(pcov)) # array([3028947.74320428, 544624.83253159, 3028947.74412785]) y_ = my_quad(x, *popt) r2 = np.corrcoef(y, y_)[0, 1] # 0.9968876754155439 # remove any one parameter def my_quad(x, a, b): return a*(1-x**2/(2*b**2)) bounds=([0, 0], [np.inf, np.inf]) popt, pcov = curve_fit(my_quad, x, y, bounds=bounds) err = np.sqrt(np.diag(pcov)) # array([0.01460553, 0.00260903]) y_ = my_quad(x, *popt) r2 = np.corrcoef(y, y_)[0, 1] # 0.9968876754155439 # plot results plt.plot(x, y, '.') plt.plot(x, y_, '-') plt.plot(x, y_2, '--')
Question: I am trying to convert a QDateTime object in pyside6 to a python datetime object. Consider the following code: from PySide6.QtCore import Qt, QDateTime, QTimeZone import datetime qdatetime = QDateTime.currentDateTime() print(qdatetime.offsetFromUtc()) qdatetime.setTimeZone(QTimeZone.UTC) print(qdatetime.toString(Qt.ISODate)) print(qdatetime.offsetFromUtc()) pt = qdatetime.toPython() print(pt) print(pt.tzinfo) ptt = pt.replace(tzinfo = datetime.timezone.utc) print(ptt.tzinfo) print(ptt)  The output looks like this: 3600 2025-01-06T18:56:43Z 0 2025-01-06 18:56:43.251000 None UTC 2025-01-06 18:56:43.251000+00:00  Obviously I can attach a timezone both to the QDateTime object and the python datetime object. The conversion however seems to delete the timezone information. The workaround would be to reattach the timezone after conversion. This seems like a complicated way to do this. Is there a way to convert without loosing timezone in the first place?

Answer: The toPython() function of PySide originates from the original toPyDateTime() function of PyQt4, and at that time the QDateTime class didn't provide time zone information. The behavior wasn't changed even with the more recent PyQt5/6 implementation, which wasn't updated to reflect the time zone info introduced since Qt 5.2. For various reasons, including the different implementation between Qt and Python, that was probably not implemented as a choice, and I sincerely doubt it will. Those helper functions were intended for simple cases, and more complex situation may require custom implementation. One way to properly convert the QDateTime while preserving the offset could be to just use the explicit date time string for datetime.fromisoformat() and adding the UTC offset from the original QDateTime: from PyQt6.QtCore import QDateTime from datetime import datetime def datetimeQt2Py(qDateTime): s = qDateTime.offsetFromUtc() if not s: return datetime.fromisoformat( qDateTime.toString(Qt.DateFormat.ISODate)) sign = '+' if s > 0 else '-' s = abs(s) h = m = 0 if s >= 60: m = s // 60 s %= 60 if m >= 60: h = m // 60 m %= 60 return datetime.fromisoformat(qDateTime.toUTC().toString( f'yyyy-MM-ddTHH:mm:ss{sign}{h:02}:{m:02}:{s:02}')) now = QDateTime.currentDateTime() dt = datetimeQt2Py(now) print(now) print(dt) print(dt.tzinfo)  The above will obviously not preserve the time zone name, nor any DST information. All that could be achieved by dynamically creating datetime.tzinfo objects.
Question: I have files associated with people scattered around different directories. I can find them with a master file. Some of them need to be pulled into my working directory. Once I've pulled them, I need to update the master file to reflect the change. To keep track of which files were moved, I record the person's name in another file. master.txt Bob "/home/a/bob.txt" Linda "/home/b/linda.txt" Joshua "/home/a/josh.txt" Sam "/home/f/sam.txt"  moved.txt Linda Sam  Expected result of master.txt Bob "/home/a/bob.txt" Linda "/workingdir/linda.txt" Joshua "/home/a/josh.txt" Sam "/workingdir/sam.txt"  I've tried grep -f moved.txt master.txt | sed "s?\/.*\/?"`pwd`"\/?" grep -f moved.txt master.txt | sed "s?\/.*\/?"`pwd`"\/?" master.txt grep -f moved.txt master.txt | sed -i "s?\/.*\/?"`pwd`"\/?"  As an added complication, this is going to execute as part of a python script, so it needs to be able to work within a subprocess.run(cmd).  Update 1: Based on some questions, here is what the relevant section of my Python code looks like. I'm trying to figure out what the next step should be in order to update the paths of the flagged files in master. commands = ['program finder.exe "flaggedfile" > master.txt' ,'sed "\#"`pwd`"#d" list.txt | sed "s/:.*//" > moved.txt' ,'program mover.exe moved.txt .' #,'cry' ] for cmd in commands: status = subprocess.run(cmd ,cwd=folder ,shell=True ,stdout=subprocess.DEVNULL ,stderr=subprocess.DEVNULL )  "program" is a program that I work with, and "finder.exe" and "mover.exe" are executables for that program, which I'm using to locate flagged files and move into the working directory.

Answer: Frame challenge: I don't need to compare the two files, I can do sed shenanigans.  Extract matching lines from master to have the entire line. Wrangle the lines to have just the path. Use sed grouping to copy the old path and filename, then build a sed command in place inside a new file. Execute the new file.  This means that the python snippet looks like: commands = ['program finder.exe "flaggedfile" > list.txt' ,'sed "\#"`pwd`"#d" list.txt | sed "s/:.*//" > moved.txt' ,'program mover.exe moved.txt .' ,'grep -f moved.txt master.txt | grep -o "\/.*\.txt" | sed -r "s?^(.*/)(.*)?sed -i \\"s#\\1\\2#`pwd`/\\2#g\\" master.txt?g" > updatemaster.txt' ,'. ./updatemaster.txt' ]  I tested this and it does work. Thank you to everyone for your advice. I understand that I have weird constraints that I'm working with, and I'm sorry that I can't use python properly because of it.
Question: I’ve a project structured as follows: . ├── hello │ ├── __init__.py │ └── animal.py ├── tests │ ├── __init__.py │ └── test_animal.py ├── README └── pyproject.toml  This is just a personal Python library, and doesn’t need to be published or distributed. The usage consists of running pytest and mypy from the root directory. Among other things, the pyproject.toml contains the following sections: [project.optional-dependencies] test = [ "pytest", ] lint = [ "ruff", "mypy", ] [tool.mypy] exclude = [ 'venv', ] ignore_errors = false warn_return_any = true disallow_untyped_defs = true  I install the dependencies locally as follows: % $(brew --prefix python)/bin/python3 -m venv ./venv % ./venv/bin/python -m pip install --upgrade pip '.[test]' '.[lint]'  But my GitHub CI fails with the following error: hello/__init__.py: error: Duplicate module named "hello" (also at "./build/lib/hello/__init__.py") hello/__init__.py: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules for more info hello/__init__.py: note: Common resolutions include: a) using `--exclude` to avoid checking one of them, b) adding `__init__.py` somewhere, c) using `--explicit-package-bases` or adjusting MYPYPATH Found 1 error in 1 file (errors prevented further checking) Error: Process completed with exit code 2.  As suggested, running mypy with --explicit-package-bases fixes this problem, but so does addition of the following section in pyproject.toml. [tool.setuptools] py-modules = []  I’ve reviewed the mypy, and setuptools documentations, but am not sure which of the two is better suited for my purpose, or why are they even necessary. As mentioned earlier, I’m not trying to publish or distribute this as a Python package. Which one of the two configurations the recommended way to go, and why?

Answer: I found a pip ticket for this exact problem where pytest was confused by the presence of a build directory. One of the suggestions in the ticket was to ignore the build directory. Apparently, in-place builds were introduced in pip 20.1, and is now the default. The following configuration in pyproject.toml solves the problem. However, it's a surprise that mypy does't exclude the directory automatically. I've created a ticket on their GitHub. [tool.mypy] exclude = [ "venv", "build", ]
Question: I use a proprietary Python package. Within this Python package the following DLL load command fails. scripting_api = ctypes.CDLL("scripting_api_interface")  Could not find module 'scripting_api_interface' (or one of its dependencies). Try using the full path with constructor syntax. I know the path to the DLL scripting_api-interface.dll and added within my Python code the following DLL path. os.environ['PATH'] = 'L:\win64' + os.pathsep  But still loading the DLL will fail. I created a test environment where I used the following command. scripting_api = ctypes.CDLL("L:\win64\scripting_api_interface.dll")  Which works as expected. But I can't change the DLL call, because it is provided by the mentioned Python package. Are there any other options to get this running?

Answer: Call CDLL with the version that works before importing the package. Once the DLL is loaded, additional loads are ignored. Example (Win11 x64)... test.c (simple DLL source, MSVC: cl /LD test.c): __declspec(dllexport) void func() {}  With test.dll in the current directory, loading will fail without an explicit path for DLLs not in the "standard" search path. The current directory is not part of the search path for security reasons. >>> import ctypes as ct >>> dll = ct.CDLL('test') Traceback (most recent call last): File "<python-input-1>", line 1, in <module> dll = ct.CDLL('test') File "C:\dev\Python313\Lib\ctypes\__init__.py", line 390, in __init__ self._handle = _dlopen(self._name, mode) ~~~~~~~^^^^^^^^^^^^^^^^^^ FileNotFoundError: Could not find module 'test' (or one of its dependencies). Try using the full path with constructor syntax. >>> dll = ct.CDLL('./test') # explicit relative path works >>> dll = ct.CDLL('test') # now without path works because already loaded >>>  So in your case the following should work: import ctypes as ct ct.CDLL('L:/win64/scripting_api_interface.dll') import your_package
Question: Qt has several IntEnum's that support custom , user-specified types or roles. A few examples are:  QtCore.Qt.ItemDataRole.UserRole QtCore.QEvent.Type.User  In both cases, a user type/role is created by choosing an integer >= the User type/role myType = QtCore.QEvent.Type.User + 1  The problem is that all of the functions that deal with these type/roles expect an instance of the IntEnum, not an int, and mypy will report an error. from PySide6.QtCore import QEvent class MyEvent(QEvent): def __init__(self) -> None: super().__init__(QEvent.Type.User + 1)  Mypy error: No overload variant of "__init__" of "QEvent" matches argument type "int"  Integrated type checking in VS code with Pylance gives a similar error: No overloads for "__init__" match the provided arguments PylancereportCallIssue QtCore.pyi(2756, 9): Overload 2 is the closest match Argument of type "int" cannot be assigned to parameter "type" of type "Type" in function "__init__" "int" is not assignable to "Type" PylancereportArgumentType  What type hinting can I do from my end to satisfy mypy? Is this something that needs to be changed in Qt type hinting?

Answer: In PySide6/PyQt6, the type of a user-defined int enum member should be preserved by using the constructor of the enum: from PySide6.QtCore import QEvent class MyEvent(QEvent): def __init__(self) -> None: super().__init__(QEvent.Type(QEvent.Type.User + 1))  Assuming the latest PySide6 stubs are installed, a file with the above contents will produce no errors when checked with the mypy command-line tool. NB: the implementation defines _missing_ to handle unknown members, and this works for all enums that subclass IntEnum, regardless of whether the values make any sense: >>> QEvent.Type.User <Type.User: 1000> >>> QEvent.Type(QEvent.Type.User + 1) <Type.1001: 1001> >>> QEvent.Type.MaxUser <Type.MaxUser: 65535> >>> QEvent.Type(QEvent.Type.MaxUser + 10) <Type.65545: 65545> >>> >>> QFrame.Shape.__members__ mappingproxy({'NoFrame': <Shape.NoFrame: 0>, 'Box': <Shape.Box: 1>, 'Panel': <Shape.Panel: 2>, 'WinPanel': <Shape.WinPanel: 3>, 'HLine': <Shape.HLine: 4>, 'VLine': <Shape.VLine: 5>, 'StyledPanel': <Shape.StyledPanel: 6>}) >>> QFrame.Shape(42) <Shape.42: 42>
Question: I have the following matrix import numpy as np A = np.array([ [0, 0, 0, 0, 1, 0, 1], [0, 0, 0, 0, 0, 0, 1], [1, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0] ]).astype(bool)  How do I fill all the rows column-wise after a column is True? My desired output:  [0, 0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1], [1, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0] 

Answer: You could use logical_or combined with accumulate: np.logical_or.accumulate(A, axis=1)  Output: array([[False, False, False, False, True, True, True], [False, False, False, False, False, False, True], [ True, True, True, True, True, True, True], [False, False, False, False, False, False, False]])  If you want integers, go with maximum: np.maximum.accumulate(A.astype(int), axis=1) array([[0, 0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1], [1, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0]])
Question: I am working with a Timeseries data wherein I am trying to perform outlier detection using IQR method. Sample Data: import pandas as pd import numpy as np df = pd.DataFrame({'datecol' : pd.date_range('2024-1-1', '2024-12-31'), 'val' : np.random.random.randin(low = 100, high = 5000, size = 8366})  my function: def is_outlier(x): iqr = x.quantile(.75) - x.quantile(.25) outlier = (x <= x.quantile(.25) - 1.5*iqr) | (x >= x.quantile(.75) + 1.5*iqr) return np.select([outlier], [1], 0) df.groupby(df['datecol'].dt.weekday)['val'].apply(is_outlier)  to which the output is something like below: 0 [1,1,0,0,.... 1 [1,0,0,0,.... 2 [1,1,0,0,.... 3 [1,0,1,0,.... 4 [1,1,0,0,.... 5 [1,1,0,0,.... 6 [1,0,0,1,....  I am expecting a single series as output which I can add back to the original dataframe as a flag column. Can someone please help me with this

Answer: You should use groupby.transform, not apply: df['flag'] = df.groupby(df['datecol'].dt.weekday)['val'].transform(is_outlier)  Alternatively, explicitly return a Series and use group_keys=False: def is_outlier(x): iqr = x.quantile(.75) - x.quantile(.25) outlier = (x <= x.quantile(.25) - 1.5*iqr) | (x >= x.quantile(.75) + 1.5*iqr) return pd.Series(np.where(outlier, 1, 0), index=x.index) df['flag'] = (df.groupby(df['datecol'].dt.weekday, group_keys=False) ['val'].apply(is_outlier) )  Note that with a single condition, np.where should be preferred to np.select. You could also use a vectorial approach with groupby.quantile: wd = df['datecol'].dt.weekday g = df.groupby(wd)['val'] q25 = g.quantile(.25) q75 = g.quantile(.75) iqr = wd.map(q75-q25) df['flag'] = 1 - df['val'].between(wd.map(q25) - 1.5*iqr, wd.map(q75) + 1.5*iqr)  Output:  datecol val flag 0 2024-01-01 3193 0 1 2024-01-02 1044 0 2 2024-01-03 2963 0 3 2024-01-04 4448 0 4 2024-01-05 1286 0 .. ... ... ... 361 2024-12-27 1531 0 362 2024-12-28 4565 0 363 2024-12-29 3396 0 364 2024-12-30 1870 0 365 2024-12-31 3818 0