eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Hello Stack OF Community, * Basically my goal is to extract values from an excel file, after reading through data from another column.* ** Thickness** of parcel, with values for example - [0.12, 0.12, 0.13, 0.14, 0.14, 0.15] (Heading: Thickness (mm)) Weight of parcel, with values for example - [4.000, 3.500, 2.500, 4.500, 5.000, 2.000] (Heading: Weight (KG)) Excel File: Thickness Weight 0.12 4.000 0.12 3.500 0.13 2.500 0.14 4.500 0.14 5.000 0.15 2.000 Looking to generate this using Python: Thickness Weight Parcels 0.12 7.500 2 Parcels 0.13 2.500 1 Parcels 0.14 9.500 2 Parcels 0.15 2.000 1 Parcels TOTAL: 21.500 6 Parcels The user will be shown all the current values of Thickness Available and will be allowed to input a single thickness value to get its weight or a range and get its weight. So anyone of you who can recommend me how can this task be accomplished easily and efficiently. I would be very grateful for your advice. Please note: I have only done Python Programming Language. Thank You. I have learned Openpyxl but also got to know that Pandas is an efficent tool for Data Analysis, so please let me know! Arigato!

Answer: Pandas actually uses openpyxl as well as well as some other engines inside. You can check engines field in the documentation. I think that reading and manipulations are easier with pandas, but if you need some advanced formatting, you will need to use openpyxl directly. (For basic cases pandas is enough). Here is a basic example for your problem. You will need to change formatting for you needs. import pandas as pd # uncomment to read the file # df = pd.read_excel('tmp.xlsx', index_col=None) df = pd.DataFrame({ "Thikness": [0.12, 0.12, 0.13, 0.14, 0.14, 0.15], "Weight": [4.000, 3.500, 2.500, 4.500, 5.000, 2.000, ], }) res = df.groupby(["Thikness"], as_index=False).agg( Weight=('Weight', sum), Count=('Weight', 'count'), ) # write excel writer = pd.ExcelWriter('tmp.xlsx', engine='xlsxwriter') res.to_excel(writer, sheet_name='Sheet1')
Question: I have a Python project which uses mypy for type checking. The root of my project contains a setup.py and the package folder rise, along with a virtual environment folder venv. Both my shells and VSCode are set to use this virtual environment. Most of the time, this setup works great: VSCode runs mypy every time I save and marks the errors with squiggles and in the Problems window. But for some reason, if I install the rise package in editable mode using pip install -e ., type checking in VSCode never finds any type errors. This does not happen if I install it in non-editable mode with pip install ., and if I uninstall the editable-mode package and reload the VSCode window, it immediately starts working normally again.  If I go into the Python output in the Output window, I see: > ./venv/bin/python ~/.vscode/extensions/ms-python.python-2022.18.2/pythonFiles/linter.py -m mypy --show-column-numbers --strict ./rise/action.py [...output from pylint, which is also configured and is working normally, clipped] ##########Linting Output - mypy########## Success: no issues found in 1 source file  Yet if I copy and paste the command it's echoing into the terminal, there is in fact an error: $ ./venv/bin/python ~/.vscode/extensions/ms-python.python-2022.18.2/pythonFiles/linter.py -m mypy --show-column-numbers --strict ./rise/action.py rise/action.py:140:12: error: Incompatible return value type (got "None", expected "bool") [return-value]  I'm stumped as to how to troubleshoot this further, and I haven't been able to find any reference to this issue anywhere on the web.

Answer: I ended up installing the Mypy extension for Visual Studio Code and removing mypy from my linters configuration. The extension doesn't suffer from this bug (or misconfiguration, or whatever it is), and it runs faster to boot.
Question: The following code presents a way to add two traces to a Plotly figure: import plotly.graph_objs as go fig = go.Figure() fig.add_trace(go.Scatter( x = [0, 1, 2, 3], y = [1, 2, 3, 4], mode = 'lines+markers', name = "Trace 0", )) fig.add_trace(go.Scatter( x = [5,6,7,8], y = [1, 2, 3, 4], mode = 'lines+markers', name = "Trace 1", )) fig.show()  Which looks like this:  Is it possible to merge these two traces into a single one such that they appear under the same title in the legend and share the same visual properties (i.e, color, markers, etc)? More over, merging these traces should enable toggling the visibility when clicking the title in the legend.

Answer: The other answer here is excellent, but I'll post an alternative solution for those interested. If for some reason you don't want to add another column to your dataframe (or maybe you're not using dataframes), you can specify the color of each trace and put your traces in the same legend group to ensure they toggle together. This is also a bit closer to your original code syntax wise, but it is bit more of a workaround stylistically. import plotly.graph_objs as go fig = go.Figure() trace_color = "#636EFA" ## default plotly blue fig.add_trace(go.Scatter( x = [0, 1, 2, 3], y = [1, 2, 3, 4], mode = 'lines+markers', name = "Trace 0", marker = dict(color = trace_color), showlegend = False, legendgroup = "Trace", )) fig.add_trace(go.Scatter( x = [5,6,7,8], y = [1, 2, 3, 4], mode = 'lines+markers', name = "Trace", marker = dict(color = trace_color), legendgroup = "Trace", )) fig.show()
Question: If I have table with dates in format MM/DD/YYYY like below. +---+-----------+----------+ | id| startdate| enddate| +---+-----------+----------+ | 1| 01/01/2022|01/31/2022| | 1| 02/01/2022|02/28/2022| | 1| 03/01/2022|03/31/2022| | 2| 01/01/2022|03/01/2022| | 2| 03/05/2022|03/31/2022| | 2| 04/01/2022|04/05/2022| +---+-----------+----------+  How to I group based on id column and if start and end date is continuous? One thing is if there is more than a one day gap then keep the row on a new line so the above table will become: +---+-----------+----------+ | id| startdate| enddate| +---+-----------+----------+ | 1| 01/01/2022|31/03/2022| | 2| 01/01/2022|03/01/2022| | 2| 03/05/2022|04/05/2022| +---+-----------+----------+  id = 1 becomes one row as all dates for id =1 is continuous i.e no gap > 1 but id 2 has two rows as there is a gap between 03/01/2022 and 03/05/2022.

Answer: This is a particular case of the sessionization problem (i.e. identify sessions in data based on some conditions). Here is a possible solution that uses windows. The logic behind the solution:  Associate at each row the temporally previous enddate with the same id Calculate the difference in days between each startdate and the previous enddate Identify all the rows that don't have a previous row or is at least two days after the previous row Associate at each row a session_index, that is the number of new sessions seen up to this line Aggregate grouping by id and session_index  w = Window.partitionBy("id")\ .orderBy("startdate") df = df \ .select( F.col("id"), F.to_date("startdate", "MM/dd/yyyy").alias("startdate"), F.to_date("enddate", "MM/dd/yyyy").alias("enddate") ) \ .withColumn("previous_enddate", F.lag('enddate', offset=1).over(w)) \ .withColumn("date_diff", F.datediff(F.col("startdate"), F.col("previous_enddate"))) \ .withColumn("is_new_session", F.col("date_diff").isNull() | (F.col("date_diff") > 1)) \ .withColumn("session_index", F.sum(F.col("is_new_session").cast("int")).over(w)) df.groupBy("id", "session_index") \ .agg( F.min("startdate").alias("startdate"), F.max("enddate").alias("enddate") ) \ .drop("session_index")
Question: I work on a project where we want to have multilingual site. We start with two languages defined in settings.py LANGUAGES = ( ("en-us", _("United States")), ("cs", _("Czech Republic")), )  I am not the programmer doing the work but if I understood correctly all we need is to be able to add - for example - French language for the whole website but not via setting.py but Django admin web interface. LANGUAGES = ( ("en-us", _("United States")), ("cs", _("Czech Republic")), ("fr", _("French")), )  We are using rosetta for translating in Django admin. So I want to use Django admin to add new laguage so it appears in rosetta interface. Could someone tell me how we can control ( add or remove or disable ) languages from Django admin? I checked these but did not find the answer  Adding new site language in Django admin How to manage system languages from django admin site? https://djangowaves.com/tutorial/multiple-languages-in-Django/ Add translation for model field using django rosetta Adding languages dynamically through Django Admin 

Answer: The short answer is that you can't do that. The settings.py of a Django project is not designed, and not recommended to be modified by the web application.(It can introduce a security breach.) So I recommend to change LANGUAGES manually, or to enable all languages supported by Django by removing LANGUAGES key. Of course, don't forget to generate message files with the makemessages command. If you really want such a dynamic feature, your best bet will be to implement it on your own by modifying the Django Rosetta source code.(Define a preference item for supported languages on a DB model, and filter languages by its value.)
Question: Suppose I have a program that looks like this: jobs = [list_of_values_to_consume_and_act] with multiprocessing.Pool(8) as pool: results = pool.map(func, jobs)  And whatever is done in func can raise an exception due to external circumstances, so I can't prevent an exception from happening. How will the pool behave on exception?  Will it only terminate the process that raised an exception and let other processes run and consume the jobs?  If yes, will it start another process to pick up the slack?  What about the job being handled by the dead process, will it be 'resubmitted' to the pool?  In any case, how do I 'retrieve' the exception?  

Answer:  No processes will be terminated at all. All calls to the target functions from within the pool's processes are wrapped in a try...except block. Incase an exception is caught, the process informs the appropriate handler thread in the main process which passes the exception forward so it can be re-rasied. Whether or not other jobs will execute depends on if the pool is still open. Incase you do not catch this re-raised exception, the main process (or the process that started the pool) will exit, automatically cleaning up open resources like the pool (so no tasks can be executed now since pool closed). But if you catch the exception and let the main process continue running then the pool will not shutdown and other jobs will execute as scheduled. N/A The outcome of a job is irrelevant, once it's run once by any process, that job is marked completed and not resubmitted to the pool. Wrap your call to pool.map in a try...except block? Do note that incase one of your jobs do raise an error, then the results of other successful jobs will become inaccessible as well (because these are stored after the call to pool.map completes, but the call never successfully completed). In such cases, where you need to catch exceptions of individual jobs, it's better to use pool.imap or pool.apply_async  Example of catching exception for individual tasks using imap: import multiprocessing import time def prt(value): if value == 3: raise ValueError(f"Error for value {value}") time.sleep(1) return value if __name__ == "__main__": with multiprocessing.Pool(3) as pool: jobs = pool.imap(prt, range(1, 10)) results = [] for i in range(10): try: result = next(jobs) except ValueError as e: print(e) results.append("N/A") # This means that this individual task was unsuccessful except StopIteration: break else: results.append(result) print(results)  Example of catching exception for individual tasks using apply_async import multiprocessing import time def prt(value): if value == 3: raise ValueError(f"Error for value {value}") time.sleep(1) return value if __name__ == "__main__": pool = multiprocessing.Pool(3) job = [pool.apply_async(prt, (i,)) for i in range(1, 10)] results = [] for j in job: try: results.append(j.get()) except ValueError as e: print(e) results.append("N/A") print(results)
Question: Why does Python implement dataclasses.dataclass as a class decorator and not as a base class? I think it would be at least clearer from the conceptual point of view to have it as a base class: the __init__ method seems to be the only thing a dataclass decorator adds to a class, and adding methods and attributes is what any straightforward base class in usually intended to do. Why to implement a decorator that intrinsically modifies a class? Base classes are just meant for this. Also, having a "Dataclass" base class would make easier for users to modify its behaviour in case any particular working mechanism is needed, one would just have to overwrite base class' methods when inheriting the dataclass. Since it clearly has been made this way for some reason I'm trying to figure out why. The only thing that comes to my mind might be some performance-related thing, I think inheriting a class should be slower that just passing a class through a function, however I'm not sure dataclasses are meant to be highly performant - nor the Python language itself - and in any case for that we have named tuples.

Answer: Dataclasses were introduced in PEP 557, which describes some of the design considerations for this feature, including rejected ideas. However, there is no mention of any rejected alternatives to using a decorator, such as using a base class instead. So it seems we cannot give a definitive answer for why a decorator was used rather than a base class. Perhaps using a base class simply wasn't considered as an option. That said, it is not correct to say that the @dataclass decorator only adds the __init__ method. The signature of the dataclass function shows that there are several other features provided, which you can opt in or out of by passing boolean flags:  __eq__, enabled by default __repr__, enabled by default Ordering methods such as __lt__, opt-in __hash__, opt-in Immutability through the __setattr__ and __delattr__ methods, opt-in  In principle, these methods could be implemented in a generic way in a base class. However, there is a potential for greater performance by generating code for these methods once the field specifications are known; this way, you don't have to look at those field specifications on every method call. (These methods are generated when the dataclass is declared, by building Python code as a string and calling exec, see here.) So, one big advantage of using a decorator is that you can look at the field specifications just once and then generate code for that specific dataclass. By analogy, implementing these in a generic way in a base class is like interpreting, whereas generating code with a decorator is like compiling. (Of course the resulting generated code is still interpreted by the CPython interpreter, which is why this is only an analogy.) Or, putting it a different way, it makes less sense to inherit methods such as __eq__, __lt__ and __hash__ from a base class, because these methods don't do the same thing for every dataclass, except in a much more generic sense.
Question: I want to import a JSON lines file into pandas. I tried to import it like a regular JSON file, but it did not work: js = pd.read_json (r'C:\Users\Name\Downloads\profilenotes.jsonl') 

Answer: This medium article provides a fairly simple answer, which can be adapted to be even shorter. All you need to do is read each line then parse each line with json.loads(). Like this: import json import pandas as pd lines = [] with open(r'test.jsonl') as f: lines = f.read().splitlines() line_dicts = [json.loads(line) for line in lines] df_final = pd.DataFrame(line_dicts) print(df_final)  As cgobat pointed out in a comment, the medium article adds a few extra unnecessary steps, which have been optimized in this answer.
Question: How to install the latest python version 3.11.0 on ArchLinux through pacman? ArchLinux wiki says current version is 3.10, although python 3.11 has been officially released. When running sudo pacman -Syyu p I'm welcomed with warning: python-3.10.8-3 is up to date. Am I doing something wrong?

Answer: Use AUR like "yay" to get the new python3.11. If you haven't installed yay on your system, setup yay by following these instructions Run this command after setting up yay in your system: yay -S python311
Question: I have a Pandas dataframe with the following structure: A B C a b 1 a b 2 a b 3 c d 7 c d 8 c d 5 c d 6 c d 3 e b 4 e b 3 e b 2 e b 1  And I will like to transform it into this: A B C1 C2 C3 C4 C5 a b 1 2 3 NAN NAN c d 7 8 5 6 3 e b 4 3 2 1 NAN  In other words, something like groupby A and B and expand C into different columns. Knowing that the length of each group is different. C is already ordered Shorter groups can have NAN or NULL values (empty), it does not matter.

Answer: Use GroupBy.cumcount and pandas.Series.add with 1, to start naming the new columns from 1 onwards, then pass this to DataFrame.pivot, and add DataFrame.add_prefix to rename the columns (C1, C2, C3, etc...). Finally use DataFrame.rename_axis to remove the indexes original name ('g') and transform the MultiIndex into columns by using DataFrame.reset_indexcolumns A,B: df['g'] = df.groupby(['A','B']).cumcount().add(1) df = df.pivot(['A','B'], 'g', 'C').add_prefix('C').rename_axis(columns=None).reset_index() print (df) A B C1 C2 C3 C4 C5 0 a b 1.0 2.0 3.0 NaN NaN 1 c d 7.0 8.0 5.0 6.0 3.0 2 e b 4.0 3.0 2.0 1.0 NaN  Because NaN is by default of type float, if you need the columns dtype to be integers add DataFrame.astype with Int64: df['g'] = df.groupby(['A','B']).cumcount().add(1) df = (df.pivot(['A','B'], 'g', 'C') .add_prefix('C') .astype('Int64') .rename_axis(columns=None) .reset_index()) print (df) A B C1 C2 C3 C4 C5 0 a b 1 2 3 <NA> <NA> 1 c d 7 8 5 6 3 2 e b 4 3 2 1 <NA>  EDIT: If there's a maximum N new columns to be added, it means that A,B are duplicated. Therefore, it will beneeded to add helper groups g1, g2 with integer and modulo division, adding a new level in index: N = 4 g = df.groupby(['A','B']).cumcount() df['g1'], df['g2'] = g // N, (g % N) + 1 df = (df.pivot(['A','B','g1'], 'g2', 'C') .add_prefix('C') .droplevel(-1) .rename_axis(columns=None) .reset_index()) print (df) A B C1 C2 C3 C4 0 a b 1.0 2.0 3.0 NaN 1 c d 7.0 8.0 5.0 6.0 2 c d 3.0 NaN NaN NaN 3 e b 4.0 3.0 2.0 1.0
Question: Tuple in python is immutable by design, so if we try to mutate a tuple object, python emits following TypeError which make sense. >>> a = (1, 2, 3) >>> a[0] = 12 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'tuple' object does not support item assignment  So my question is, if tuple is immutable by design why cpython exposes PyTuple_SetItem as C-API?. From the documentation it's described as  int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o) Insert a reference to object o at position pos of the tuple pointed to by p. Return 0 on success. If pos is out of bounds, return -1 and set an IndexError exception.  Isn't this statement exactly equal to tuple[index] = value in python layer?. If the goal was to create a tuple from collection of items we could have use PyTuple_Pack. Additional note: After lot of trial and error with ctypes.pythonapi I managed to mutate tuple object using PyTuple_SetItem import ctypes from ctypes import py_object my_tuple = (1, 2, 3) newObj = py_object(my_tuple) m = "hello" # I don't know why I need to Py_DecRef here. # Although to reproduce this in your system, no of times you have # to do `Py_DecRef` depends on no of ref count of `newObj` in your system. ctypes.pythonapi.Py_DecRef(newObj) ctypes.pythonapi.Py_DecRef(newObj) ctypes.pythonapi.Py_DecRef(newObj) ctypes.pythonapi.Py_IncRef(m) PyTuple_SetItem = ctypes.pythonapi.PyTuple_SetItem PyTuple_SetItem.argtypes = ctypes.py_object, ctypes.c_size_t, ctypes.py_object PyTuple_SetItem(newObj, 0, m) print(my_tuple) # this will print `('hello', 2, 3)` 

Answer: Similarly, there is a PyTuple_Resize function with the warning  Because tuples are supposed to be immutable, this should only be used if there is only one reference to the object. Do not use this if the tuple may already be known to some other part of the code. The tuple will always grow or shrink at the end. Think of this as destroying the old tuple and creating a new one, only more efficiently.  Looking at the source, there is a guard on the function if (!PyTuple_Check(op) || Py_REFCNT(op) != 1) { .... error ....  Sure enough, this is only allowed when there is only 1 reference to the tuple - that reference being the thing that thinks its a good idea to change it. So, a tuple is "mostly immutable" but C code can change it in limited circumstances to avoid the penalty of creating a new tuple.
Question: Out of the box, Typer CLIs only recognize the long help option --help to display the help text. I would like to also accept the short option -h but I can't figure out how. I've searched the docs to no avail. Do I need to alias -h to --help and if so, how do I do that?

Answer: The key is to use context_settings={"help_option_names": ["-h", "--help"]}) As suggested by @jvx8ss in the comments, one needs to convert a typer.run app to one using @app.command() decorators. Here is a minimal working example: import typer app = typer.Typer(context_settings={"help_option_names": ["-h", "--help"]}) @app.command() def main(name: str): print(f"Hello {name}") if __name__ == "__main__": app()
Question: I'm unable to use polars dataframes with scikit-learn for ML training. Currently, I'm preprocessing all dataframes in polars and convert them to pandas for model training in order for it to work. Is there any method to directly use polars dataframes with the scikit-learn API (without converting to pandas first)?

Answer: You must call to_numpy when passing a DataFrame to sklearn. Though sometimes sklearn can work on polars Series it is still good type hygiene to transform to the type the host library expects. import polars as pl from sklearn.linear_model import LinearRegression data = pl.DataFrame( np.random.randn(100, 5) ) x = data.select([ pl.all().exclude("column_0"), ]) y = data.select(pl.col("column_0").alias("y")) x_train = x[:80] y_train = y[:80] x_test = x[80:] y_test = y[80:] m = LinearRegression() m.fit(X=x_train.to_numpy(), y=y_train.to_numpy()) m.predict(x_test.to_numpy())
Question: Flake8 was installed lately by one of the updates of vscode. I think it is time to comply to the "rules" of python to writer better and more readable code. Unfortunately I have some errors that I cannot fix in the code (no discussion about that, but a local module has to be loaded before some others). I want to ignore some warning of Flake8. I have the following settings in settings.json:  "python.linting.flake8Enabled": true, "python.linting.enabled": true, "python.linting.flake8Args": [ "--extend-ignore=E203,E266,E501,W503,E402", "--max-line-length=98" ],  Both the warnings are not ignored and the maximum line length has not changed. In the GUI the settings are also visible (python > Linting > Flake8 Args). EDIT: I also tried (what was suggested by rzlvmp below) "--ignore=E203,E266,E501,W503,E402"  And I restarted vscode and sometime the whole computer to be sure.

Answer: seems like python.linting.flake8Args no longer works, I can get flake to work, but I get everything. My solution was to install the flake8 plugin: https://marketplace.visualstudio.com/items?itemName=ms-python.flake8 and use the flake8.args: "flake8.args": [ "--ignore=E24,E128,E201,E202,E225,E231,E252,E265,E302,E303,E401,E402,E501,E731,W504,W605", "--verbose" ],