eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'd really like to use Python enum types for this model I'm working on. The problem is one of the potential values from the data provider is an empty string. Given this model... from sqlalchemy import Column, Enum class Events(Base): __tablename__ = 'events' ... restriction_code = Column(Enum(RestrictionCode)) ...  ...one of the enum types would have an "empty string" as an attribute like this... from enum import Enum class RestrictionCode(Enum): A = 'A Description' B = 'B Description' C = 'C Description' D = 'D Description' '' = 'No Restriction'  The above produces the not surprising: SyntaxError: can't assign to literal. Is this even possible or is there a workaround?

Answer: No, each Enum member must have a name. The intended way to use Enums in this scenario would be to have the value of the member be the value stored in the database, so your Enum should look like: class RestrictionCode(Enum): A = 'A' B = 'B' C = 'C' D = 'D' NONE = ''  If you want a description as well, you'll need to design your own Enum -- but it's not hard: class DescriptiveEnum(Enum): # def __new__(cls, value, description): member = object.__new__(cls) member._value_ = value member.description = description return member  Then your RestrictionCode will look like: class RestrictionCode(DescriptiveEnum): A = 'A', 'A description' B = 'B', 'B description' C = 'C', 'C description' D = 'D', 'D description' NONE = '', 'No restriction'
Question: I recently tested many hyperparameter combinations using sklearn.model_selection.GridSearchCV. I want to know if there is a way to call all previous estimators that were trained in the process. search = GridSearchCV(estimator=my_estimator, param_grid=parameters) # `my_estimator` is a gradient boosting classifier object # `parameters` is a dictionary containing all the hyperparameters I want to try  I know I can call the best estimator with search.best_estimator_, but I would like to call all other estimators as well so I can test their individual performance. The search took around 35 hours to complete, so I really hope I do not have to do this all over again. NOTE: This was asked a few years ago (here), but sklearn has been updated multiple times since and the anwer may be different now (I hope).

Answer: No, none of the tested models are saved, except (optionally, but by default) one final one trained on the entire training set, your best_estimator_. Especially when models store significant amounts of data (e.g. KNNs), saving all the fitted estimators would be very memory-expensive, and usually not of much use. (cross_validate does have a parameter return_estimator, but the hyperparameter tuners do not. If you have a compelling reason to add it, it probably wouldn't take much work and you could open a GitHub Issue at sklearn.) However, you do have the cv_results_ attribute that documents the scores of all of the tested estimators. That's usually enough for inspection purposes.
Question: Let's say I put a breakpoint in the first line. I see no option to simply skip the 2nd line and jump straight to the print statement. Is there any hidden option? If not, what is the most non-intrusive way? Commenting out the lines I don't wanna run is not elegant. a = 3 a = 4 print(a) 

Answer: You can do right click on the third statement and Jump to Cursor. This is a manual action though... I don't think there is a mode to only run breakpointed lines...
Question: I need to wait until a certain condition/equation becomes True in a asynchronous function in python. Basically it is a flag variable which would be flagged by a coroutine running in asyncio.create_task(). I want to await until it is flagged in the main loop of asyncio. Here's my current code: import asyncio flag = False async def bg_tsk(): await asyncio.sleep(10) flag = True async def waiter(): asyncio.create_task(bg_tsk()) await asyncio.sleep(0) # I find the below part unpythonic while not flag: await asyncio.sleep(0.2) asyncio.run(waiter())  Is there any better implementation of this? Or is this the best way possible? I have tried using 'asyncio.Event', but it doesnt seem to work with 'asyncio.create_task()'.

Answer: Using of asyncio.Event is quite straightforward. Sample below. Note: Event should be created from inside coroutine for correct work. import asyncio async def bg_tsk(flag): await asyncio.sleep(3) flag.set() async def waiter(): flag = asyncio.Event() asyncio.create_task(bg_tsk(flag)) await flag.wait() print("After waiting") asyncio.run(waiter())
Question: Wondering how to print the range of dates in a dataframe. Seems like it would be very simple but I can't find answers anywhere. Is there an easy way to do this with pandas datetime module? So if this was a small version of the dataframe for example:     Date Id Value     2020-09-23 14:00:00 4752764 12212   2020-10-25 08:00:00 4752764 12298   2020-10-28 12:00:00 4752764 12291   2020-10-29 18:00:00 4752764 12295     How could I get an output like: date_range = 2020-09-23 to 2020-10-29  OR date_range = 23rd of September, 2020 to 29th of October, 2020  I appreciate any answers :)

Answer: Try this df['Date'] = pd.to_datetime(df['Date']) # If your Date column is of the type object otherwise skip this date_range = str(df['Date'].dt.date.min()) + ' to ' +str(df['Date'].dt.date.max())
Question: How to check from within a model if it is currently in train or eval mode?

Answer: From the Pytorch forum, with a small tweak: use if self.training: # it's in train mode else: # it's in eval mode  Always better to have a stack overflow answer than to look at forums. Explanation about the modes
Question: In the paper Attention is all you need, under section 5.3, the authors suggested to increase the learning rate linearly and then decrease proportionally to the inverse square root of steps.  How do we implement this in PyTorch with Adam optimizer? Preferably without additional packages.

Answer: PyTorch provides learning-rate-schedulers for implementing various methods of adjusting the learning rate during the training process. Some simple LR-schedulers are are already implemented and can be found here: https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate In your special case you can - just like the other LR-schedulers do - subclass _LRScheduler for implementing a variable schedule based on the number of epochs. For a bare-bones method you only need to implement __init__() and get_lr() methods. Just note that many of these schedulers expect you to call .step() once per epoch. But you can also update it more frequently or even pass a custom argument just like in the cosine-annealing LR-scheduler: https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CosineAnnealingLR
Question: In the Indirect parametrization example I want to type hint request.param indicating a specific type, a str for example. The problem is since the argument to fixt must be the request fixture there seems to be no way to indicate what type the parameters passed through the "optional param attribute" should be (quoting the documentation). What are the alternatives? Perhaps documenting the type hint in the fixt docstring, or in the test_indirect docstring? @pytest.fixture def fixt(request): return request.param * 3 @pytest.mark.parametrize("fixt", ["a", "b"], indirect=True) def test_indirect(fixt): assert len(fixt) == 3 

Answer: As of now (version 6.2), pytest doesn't provide any type hint for the param attribute. If you need to just type param regardless of the rest of FixtureRequest fields and methods, you can inline your own impl stub: from typing import TYPE_CHECKING if TYPE_CHECKING: class FixtureRequest: param: str else: from typing import Any FixtureRequest = Any @pytest.fixture def fixt(request: FixtureRequest) -> str: return request.param * 3  If you want to extend existing typing of FixtureRequest, the stubbing gets somewhat more complex: from typing import TYPE_CHECKING if TYPE_CHECKING: from pytest import FixtureRequest as __FixtureRequest class FixtureRequest(__FixtureRequest): param: str else: from pytest import FixtureRequest @pytest.fixture def fixt(request: FixtureRequest) -> str: return request.param * 3  Ideally, pytest would allow generic param types in FixtureRequest, e.g. P = TypeVar("P") # generic param type class FixtureRequest(Generic[P]): def __init__(self, param: P, ...): ...  You would then just do from pytest import FixtureRequest @pytest.fixture def fixt(request: FixtureRequest[str]) -> str: return request.param * 3  Not sure, however, whether this typing is agreeable with the current pytest's codebase - I guess it was omitted for a reason...
Question: I hope it wasn't asked before, at least I couldn't find. I'm trying to exclude rows where Key column does not contain 'sd' value. Below is the working example for when it contains. values = [("sd123","2"),("kd123","1")] columns = ['Key', 'V1'] df2 = spark.createDataFrame(values, columns) df2.where(F.col('Key').contains('sd')).show()  how to do the opposite?

Answer: Use ~ as bitwise NOT: df2.where(~F.col('Key').contains('sd')).show()
Question: I use Windows 7 32 bit and Python 3.7. I was trying to install a module with pip and this error came up: cd C:\Windows\System32 pip install pyttsx3  Output: Traceback (most recent call last): File "d:\python\python 3.7\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "d:\python\python 3.7\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "D:\Python\Python 3.7\Scripts\pip.exe\__main__.py", line 5, in <module> ModuleNotFoundError: No module named 'pip'  How can I fix this?

Answer: Could you try? pip3 install pyttsx3
Question: Let's say I'm setting up a script or library that has a few dependencies which use Python's standard library logging module but I want to use loguru to capture all logs. My first naive attempt was a complete failure, but I'm not sure how to proceed. To test I have two files main.py: from loguru import logger from base_log import test_func if __name__ == "__main__": logger.debug("In main") test_func()  and base_log.py: import logging logger = logging.getLogger(__name__) def test_func(): logger.warning("In test_func")  If I run main.py (i.e. python main.py) then I get the following output: 2020-12-16 10:57:48.269 | DEBUG | __main__:<module>:6 - In main In test_func  When I expect: 2020-12-16 11:01:34.408 | DEBUG | __main__:<module>:6 - In main 2020-12-16 11:01:34.408 | WARNING | base_log:test_func:9 - In test_func 

Answer: You can use a custom handler to intercept standard logging messages toward your Loguru sinks as documented here. main.py will then look something like this: import logging from loguru import logger from base_log import test_func class InterceptHandler(logging.Handler): def emit(self, record): try: level = logger.level(record.levelname).name except ValueError: level = record.levelno frame, depth = logging.currentframe(), 2 while frame.f_code.co_filename == logging.__file__: frame = frame.f_back depth += 1 logger.opt(depth=depth, exception=record.exc_info).log( level, record.getMessage() ) if __name__ == "__main__": logging.basicConfig(handlers=[InterceptHandler()], level=0) logger.debug("In main") test_func()  Output: 2020-12-16 22:15:55.337 | DEBUG | __main__:<module>:26 - In main 2020-12-16 22:15:55.337 | WARNING | base_log:test_func:7 - In test_func  This should work for all well-behaved libraries that do not add any handlers other than NullHandler to the libraries' loggers. The rest may need extra work.
Question: I have a data frame: user_id url 111 google.com 111 youtube.com 111 youtube.com 111 google.com 111 stackoverflow.com 111 google.com 222 twitter.com 222 google.com 222 twitter.com  I want to create a column that will show the fact of visiting this URL before. Desired output: user_id url target 111 google.com 0 111 youtube.com 0 111 youtube.com 1 111 google.com 1 111 stackoverflow.com 0 111 google.com 1 222 twitter.com 0 222 google.com 0 222 twitter.com 1  I can do this with a loop but it doesn't look good. Is it possible to make it with pandas?

Answer: Use duplicated: df['target'] = df.duplicated().astype(int) print(df)  Output  user_id url target 0 111 google.com 0 1 111 youtube.com 0 2 111 youtube.com 1 3 111 google.com 1 4 111 stackoverflow.com 0 5 111 google.com 1 6 222 twitter.com 0 7 222 google.com 0 8 222 twitter.com 1
Question: I have this array of strings that I have split element wise so I can do stuff to it. Now I want to return them back into a sentence. my_array = (['T', 'e', 's', 't', ' ', 'w', 'o', 'r', 'd', 's', '!'])  Is there a way to join them togeter back into a sentence whilst keeping the formatting? Ideal output would be something similar to the following: joined_array = (['Test Words!']) 

Answer: Try join: my_array = (['T', 'e', 's', 't', ' ', 'w', 'o', 'r', 'd', 's', '!']) joined_array = ''.join(my_array)  Which gives: 'Test words!'
Question: I have been working on formatting a log file and finally I have arrived to the following dataframe sample, where the categories and numbers I want to add are in the same column: df = pd.DataFrame(dict(a=['Cat. A',1,1,3,'Cat. A',2,2,'Cat. B',3,5,2,6,'Cat. B',1,'Cat. C',4])) >>> a 0 Cat. A 1 1 2 1 3 3 4 Cat. A 5 2 6 2 7 Cat. B 8 3 9 5 10 2 11 6 12 Cat. B 13 1 14 Cat. C 15 4  If I sum all the numbers below each category I want to obtain: Cat. A= 1+1+3+2+2 = 9 Cat. B= 3+5+2+6+1 = 17 Cat. C= 4  I know how to do with going through all the files in the classic way, but I would like to know how to do it in a most pythonic way, considering that the numbers of rows for each category can be variable, and that the numbers of times that the category appears in each dataframe can be different too.

Answer: We can use pd.to_numeric to mark non-numeric fields as nan using Series.mask and Series.notna then use for group. Then use GroupBy.sum a = pd.to_numeric(df['a'], errors='coerce') g = df['a'].mask(a.notna()).ffill() a.groupby(g).sum() Cat. A 9.0 Cat. B 17.0 Cat. C 4.0 Name: a, dtype: float64
Question: I currently created an experiment in mlflow and created multiple runs in the experiment. from sklearn.ensemble import RandomForestRegressor from sklearn.metrics import mean_squared_error import mlflow experiment_name="experiment-1" mlflow.set_experiment(experiment_name) no_of_trees=[100,200,300] depths=[2,3,4] for trees in no_of_trees: for depth in depths: with mlflow.start_run() as run: model=RandomForestRegressor(n_estimators=trees, criterion='mse',max_depth=depth) model.fit(x_train, y_train) predictions=model.predict(x_cv) mlflow.log_metric('rmse',mean_squared_error(y_cv, predictions))  after creating the runs, I wanted to get the best run_id for this experiment. for now, I can get the best run by looking at the UI of mlflow but how can we do right the program?

Answer: we can get the experiment id from the experiment name and we can use python API to get the best runs. experiment_name = "experiment-1" current_experiment=dict(mlflow.get_experiment_by_name(experiment_name)) experiment_id=current_experiment['experiment_id']  By using the experiment id, we can get all the runs and we can sort them based on metrics like below. In the below code, rmse is my metric name (so it may be different for you based on metric name) df = mlflow.search_runs([experiment_id], order_by=["metrics.rmse DESC"]) best_run_id = df.loc[0,'run_id']
Question: I have a new install of VS Code Version 1.50.1 with the python extension that now added the Jupyter extension. The Jupyter extension build number is 2020.12.411183115 When I press shift enter on the default it adds a new line below. You can see in the video that shift + enter should work to run the line. At this point the only way I can run code in the interactive window is with ctrl + shift + p and select run selected cell. Edit after working with Danny Varod and the comments below: Changing the keyboard shortcut to ctrl + enter and nothing happens (it does not add a new line below). I press ctrl + shift + p and I see that the correct shortcut is crtl + enter but it will not trigger the action  Here is a screenshot of my keyboard shortcuts before the change.  Changing my keyboard shortcuts to match the comment below. Now when I press ctrl + enter nothing happens. 

Answer: Please use the following shortcut key settings: { "key": "shift+enter", "command": "jupyter.execSelectionInteractive", "when": "editorTextFocus" },  This shortcut key is set with the use conditions, and it can be used only when it is confirmed (including the control panel is opened). Therefore, we can remove the use conditions for this shortcut key.
Question: I want to combine multiple grouped bar charts into one figure, as the image below shows. grouped bar charts in a single figure import matplotlib import matplotlib.pyplot as plt import numpy as np labels = ['G1', 'G2', 'G3'] yesterday_test1_mean = [20, 12, 23] yesterday_test2_mean = [21, 14, 25] today_test1_mean = [18, 10, 12] today_test2_mean = [13, 13, 9]  Firstly I created each grouped bar chart by plt.subplots() x = np.arange(len(labels)) width = 0.3 fig1, ax = plt.subplots() rects1 = ax.bar(x-width/2, yesterday_test1_mean, width) rects2 = ax.bar(x+width/2, yesterday_test2_mean, width) fig2, ax = plt.subplots() rects3 = ax.bar(x-width/2, today_test1_mean, width) rects4 = ax.bar(x+width/2, today_test2_mean, width)  Then, I used add_subplot in an attempt to treat fig1 and fig2 as new axes in a new figure. fig_all = plt.figure() fig1 = fig_all.add_subplot(1,2,1) fig2 = fig_all.add_subplot(1,2,2) fig_all.tight_layout() plt.show()  But it didn't work. How can I combined several grouped bar charts into a single figure? Thanks in advance.

Answer: Well, I tried something. Here's a rough result. Only thing I changed is that rather using axes, I am just using subplot as I learned over time. So with fig and axes as output, there must be a way too. But this is all I've ever used. I've not added the legend and title yet, but I guess you can try it on your own too. Here's the code with just small change: import matplotlib.pyplot as plt import numpy as np labels = ['G1', 'G2', 'G3'] yesterday_test1_mean = [20, 12, 23] yesterday_test2_mean = [21, 14, 25] today_test1_mean = [18, 10, 12] today_test2_mean = [13, 13, 9] x = np.arange(len(labels)) width = 0.3 plt.figure(figsize=(12,5)) plt.subplot(121) plt.bar(x-width/2, yesterday_test1_mean, width) plt.bar(x+width/2, yesterday_test2_mean, width) plt.subplot(122) plt.bar(x-width/2, today_test1_mean, width) plt.bar(x+width/2, today_test2_mean, width) plt.show()  And here's your initial result:  While you see the result and try some stuff on your own, let me try to add the labels and legend to it as well as you've provided in the sample image. Edit: The final output So here it is, the exact thing you're looking for:  Code: import matplotlib.pyplot as plt import numpy as np labels = ['G1', 'G2', 'G3'] yesterday_test1_mean = [20, 12, 23] yesterday_test2_mean = [21, 14, 25] today_test1_mean = [18, 10, 12] today_test2_mean = [13, 13, 9] x = np.arange(len(labels)) width = 0.3 plt.figure(figsize=(12,5)) plt.subplot(121) plt.title('Yesterday', fontsize=18) plt.bar(x-width/2, yesterday_test1_mean, width, label='test1', hatch='//', color=np.array((199, 66, 92))/255) plt.bar(x+width/2, yesterday_test2_mean, width, label='test2', color=np.array((240, 140, 58))/255) plt.xticks([0,1,2], labels, fontsize=15) plt.subplot(122) plt.title('Today', fontsize=18) plt.bar(x-width/2, today_test1_mean, width, hatch='//', color=np.array((199, 66, 92))/255) plt.bar(x+width/2, today_test2_mean, width, color=np.array((240, 140, 58))/255) plt.xticks([0,1,2], labels, fontsize=15) plt.figlegend(loc='upper right', ncol=1, labelspacing=0.5, fontsize=14, bbox_to_anchor=(1.11, 0.9)) plt.tight_layout(w_pad=6) plt.show()
Question: I am using gunicorn to run a simple HTTP server1 using e.g. 8 sync workers (processes). For practical reasons I am interested in knowing how gunicorn distributes incoming requests between these workers. Assume that all requests take the same time to complete. Is the assignment random? Round-robin? Resource-based? The command I use to run the server: gunicorn --workers 8 bind 0.0.0.0:8000 main:app   1 I'm using FastAPI but I believe this is not relevant for this question.

Answer: Gunicorn does not distribute requests. Each worker is spawned with the same LISTENERS (e.g. gunicorn.sock.TCPSocket) in Arbiter.spawn_worker(), and calls listener.accept() on its own. The assignment in the blocking OS calls to the socket's accept() method — i.e. whichever worker is later woken up by the OS kernel and given the client connection — is an OS implementation detail that, empirically, is neither round-robin nor resource-based. Reference from the docs From https://docs.gunicorn.org/en/stable/design.html:  Gunicorn is based on the pre-fork worker model. ... The master never knows anything about individual clients. All requests and responses are handled completely by worker processes.   Gunicorn relies on the operating system to provide all of the load balancing when handling requests.  Other reading  Is there any way Load balancing on a preforked multi process TCP server build by asyncio? http://www.citi.umich.edu/projects/linux-scalability/reports/accept.html
Question: I want to make my bot react to a users message when they type a certain sentence. My code to reply: await ctx.message.reply("I just replied to you")  I get the error: ctx.message has no attribute "reply"  What code can I do to make the bot reply to the message? When I say reply, I mean the same as a user can press reply on a message

Answer: To any new user here, as of the 1.6.0 discord.py-rewrite update, you are now able to reply! Every message or context now has a reply attribute. To reply, simply use await ctx.reply('Hello!')  You can also not mention the author in the reply with mention_author=False await ctx.reply('Hello!', mention_author=False)    You can also find a basic example Here
Question: I'm doing a binary classification. Whenever my prediction equals the ground truth, I find sklearn.metrics.confusion_matrix to return a single value. Isn't there a problem? from sklearn.metrics import confusion_matrix print(confusion_matrix([True, True], [True, True]) # [[2]]  I would expect something like: [[2 0] [0 0]] 

Answer: You should fill-in labels=[True, False]: from sklearn.metrics import confusion_matrix cm = confusion_matrix(y_true=[True, True], y_pred=[True, True], labels=[True, False]) print(cm) # [[2 0] # [0 0]]  Why? From the docs, the output of confusion_matrix(y_true, y_pred) is:  C: ndarray of shape (n_classes, n_classes)  The variable n_classes is either:  guessed as the number of unique values in y_true or y_pred taken from the length of optional parameters labels  In your case, because you did not fill in labels, the variable n_classes is guessed from the number of unique values in [True, True] which is 1. Hence the result.
Question: Even if numba, cython (and especially cython.inline) exist, in some cases, it would be interesting to have inline C code in Python. Is there a built-in way (in Python standard library) to have inline C code? PS: scipy.weave used to provide this, but it's Python 2 only.

Answer: Directly in the Python standard library, probably not. But it's possible to have something very close to inline C in Python with the cffi module (pip install cffi). Here is an example, inspired by this article and this question, showing how to implement a factorial function in Python + "inline" C: from cffi import FFI ffi = FFI() ffi.set_source("_test", """ long factorial(int n) { long r = n; while(n > 1) { n -= 1; r *= n; } return r; } """) ffi.cdef("""long factorial(int);""") ffi.compile() from _test import lib # import the compiled library print(lib.factorial(10)) # 3628800  Notes:  ffi.set_source(...) defines the actual C source code ffi.cdef(...) is the equivalent of the .h header file you can of course add some cleaning code after, if you don't need the compiled library at the end (however, cython.inline does the same and the compiled .pyd files are not cleaned by default, see here) this quick inline use is particularly useful during a prototyping / development phase. Once everything is ready, you can separate the build (that you do only once), and the rest of the code which imports the pre-compiled library  It seems too good to be true, but it seems to work!