eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I made an app in tkinter which helps creating tkinter app/scripts, (when I export the file it is stored as a .py) I want to run this .py script from my app so that we can preview it immediately after export. I used subprocess.run method and it works perfectly within the python app. But when I converted the app to exe with pyinstaller then the preview thing doesn't work because of a tcl version error. init.tcl: version conflict for package "Tcl": have 8.6.10, need exactly 8.6.9 version conflict for package "Tcl": have 8.6.10, need exactly 8.6.9 while executing "package require -exact Tcl 8.6.9" (file "C:/----/_MEI170162/tcl/init.tcl" line 19) invoked from within "source {C:/-----/_MEI170162/tcl/init.tcl}" ("uplevel" body line 1) invoked from within "uplevel #0 [list source $tclfile]" This probably means that Tcl wasn't installed properly.  I tried subprocess.run, subprocess.startfile, os.system, and even webbroswer.open methods to run the exported .py tkinter app, but the same error is showing. Note that I have compiled the app with an older version of python, so the tcl version is also different. In my main system the python version is set to latest one, so subprocess is using that tcl version, but I don't want to have any connection with the main app's tcl version. As the user may install different versions of python, so there will be different tcl versions too, and this error will be shown again in that case. I tried the exec(open(file).read()) method, although it works but there are issues with some exports like pyimages missing errors. Is there any other way to run a python script which is more standalone?

Answer: Finally got an answer, we can delete the tcl environment variables before running the subprocess import os import subprocess file_path = "C:/----/test.py" def run_script(): env = os.environ.copy() if 'TCL_LIBRARY' in env: del env['TCL_LIBRARY'] if 'TK_LIBRARY' in env: del env['TK_LIBRARY'] subprocess.run(["python",file_path], shell=True, env=env)
Question: I understand the zig-zag nature of the cost function when applying gradient descent, but what bothers me is that the cost started out at a low 300 only to increase to 1600 in the end. The cost function would oscillate between 300 and 4000 to end up at 1600. I thought I should get a number that is 300 or lower. I have tried changing the learning rate and all it does is still take me to 1600. I should get a cost around 300, not one that grows it. Data: square_feet = [1661.0, 871.0, 1108.0, 1453.0, 1506.0, 1100.0, 1266.0, 1514.0, 948.0, 1878.0, 1522.0, 931.0, 1475.0, 1177.0, 1844.0, 1469.0, 2155.0, 967.0, 1092.0] prices = [1350.0, 489.0, 589.0, 539.0, 775.0, 575.0, 749.0, 795.0, 644.9, 590.0, 575.0, 699.0, 999.0, 775.0, 599.0, 599.0, 895.0, 550.0, 849.0]  Both of those lists are Pandas series in the original code, but have converted them to lists here for clarity. Main: # Add starting weight and bias w_init = 5e-1 # Increase in price for every 1 square feet b_init = 200 # Starting price for the cheapest houses # Iterations and learning rate for the gradient descent algorithm iterations = 10000 alpha = 1.0e-6 # Delicate and causes a divergence if it's set too large w_final, b_final, J_hist, p_hist = gradient_descent( square_feet, prices, w_init, b_init, alpha, iterations) print(f'w: {w_final}, b: {b_final}, Costs: {J_hist}, Weight and Bias: {p_hist}')  Functions: # Cost Function to determine cumulative error between real and predicted values def cost_function(x, y, w, b): # 1) Number of training examples m = x.size cost = 0 # 2) Index the training examples and account for cost per instance for i in range(m): y_hat = w * x[i] + b cost += (y_hat-y[i])**2 cost /= 2 * m # 3) Return total cost return cost # Compute the gradient, i.e., the scalar that improves accuracy def gradient_function(x, y, w, b): m = x.size # Partial derivatives of the cost function with respect to weight and bias dj_dw = 0 dj_db = 0 for i in range(m): y_hat = w * x[i] + b dj_dw_i = (y_hat - y[i]) * x[i] dj_db_i = (y_hat - y[i]) dj_db += dj_db_i dj_dw += dj_dw_i dj_dw /= m dj_db /= m return dj_dw, dj_db def gradient_descent(x, y, w_init, b_init, learning_rate, num_iters): # Used for graphing J_history = [] p_history = [] b = b_init w = w_init for i in range(num_iters): dj_dw, dj_db = gradient_function(x, y, w, b) # Gradient # Update weight, bias w -= learning_rate * dj_dw b -= learning_rate * dj_db # Prevents resource exhaustion; unnecessary to store similar costs # Past 100000 iterations if i < 100000: J_history.append(cost_function(x, y, w, b)) p_history.append([w,b]) return w, b, J_history, p_history  I'm stumped over this issue.

Answer: In your cost_function, the average cost is inside the loop, which results in a cost value that is m times smaller than it should be, preventing the data set from converging. You should sum up all data before averaging. def cost_function(x, y, w, b): # 1) Number of training examples m = x.size cost = 0 # 2) Index the training examples and account for cost per instance for i in range(m): y_hat = w * x[i] + b cost += (y_hat-y[i])**2 cost /= 2 * m # 3) Return total cost return cost
Question: I just installed ubuntu 24. I'm a django developer and I've been working fine with Visual Studio Code, but today when I tried to open my vscode, while it was opening it closed without any error or message! :( I restarted my computer and reinstalled Vscode. But nothing that nothing:( note: I installed vscode from app center on ubuntu

Answer: I've faced the same issue. I think you've installed it from snap store. You need to install it with deb package. Uninstall the current code(remove it). Go to official website of Visual Studio Code, download the latest version and install it with command: sudo dpkg -i code_1.90.0-1717531825_amd64.deb  In fact, the command is: sudo dpkg -i <deb_file_name>.deb
Question: I have the following two DataFrames: df 100 101 102 103 104 105 106 107 108 109 0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 2 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 3 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 4 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 5 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 6 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 7 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 8 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 9 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 df2 crit1 crit2 110 a A 109 a B 108 a A 107 b B 106 b A 105 a A 104 a B 103 a A 102 b B 101 b A 100 b A 99 b A  df contains data for ten entities 100-109 and df2 describes two criteria categorizing the ten entities 100-109 (and others, in a different order). I'd like to group df on a two-level column index (crit1,crit2) with one value per combination of (crit1,crit2), being the sum of all columns with this combination. For example, the new column with the index ('a','A') would contain the sum of columns [108,105,103]. expected result: crit1 a b crit2 A B A B 0 3.0 2.0 3.0 2.0 1 19.0 15.0 10.0 11.0 2 3.0 2.0 3.0 2.0 3 3.0 2.0 3.0 2.0 4 3.0 2.0 3.0 2.0 5 3.0 2.0 3.0 2.0 6 3.0 2.0 3.0 2.0 7 3.0 2.0 3.0 2.0 8 3.0 2.0 3.0 2.0 9 3.0 2.0 3.0 2.0  To reproduce the DataFrames: import pandas as pd import numpy as np df = pd.DataFrame(np.ones((10,10)), index=np.arange(10), columns=np.arange(100,110)) df2 = pd.DataFrame(np.array([['a','A'],['a','B'],['a','A'],['b','B'],['b','A'],['a','A'],['a','B'],['a','A'],['b','B'],['b','A'],['b','A'],['b','A']]), index=np.arange(110,98,-1), columns=['crit1','crit2']) df.iloc[1] = np.arange(1,11) 

Answer: You can reindex, set_axis with MultiIndex.from_frame then groupby.sum: out = (df.reindex(columns=df2.index) .set_axis(pd.MultiIndex.from_frame(df2), axis=1) .groupby(axis=1, level=[0, 1]).sum() )  For the latest version of pandas, using the transpose as intermediate: out = (df.T .reindex(df2.index) .set_axis(pd.MultiIndex.from_frame(df2)) .groupby(level=[0, 1]).sum().T )  Output: crit1 a b crit2 A B A B 0 4.0 2.0 2.0 2.0 1 4.0 2.0 2.0 2.0 2 4.0 2.0 2.0 2.0 3 4.0 2.0 2.0 2.0 4 4.0 2.0 2.0 2.0 5 4.0 2.0 2.0 2.0 6 4.0 2.0 2.0 2.0 7 4.0 2.0 2.0 2.0 8 4.0 2.0 2.0 2.0 9 4.0 2.0 2.0 2.0  Alternatively, pre-aggregating the indices, then building the output with concat: groups = df2.reset_index().groupby(list(df2))['index'].agg(list) out = pd.concat({k: df.reindex(columns=lst).sum(axis=1) for k, lst in groups.items()}, axis=1)  Output:  a b A B A B 0 4.0 2.0 2.0 2.0 1 4.0 2.0 2.0 2.0 2 4.0 2.0 2.0 2.0 3 4.0 2.0 2.0 2.0 4 4.0 2.0 2.0 2.0 5 4.0 2.0 2.0 2.0 6 4.0 2.0 2.0 2.0 7 4.0 2.0 2.0 2.0 8 4.0 2.0 2.0 2.0 9 4.0 2.0 2.0 2.0  Intermediate groups: crit1 crit2 a A [100, 102, 105, 107] B [101, 106] b A [104, 109] B [103, 108] Name: index, dtype: object
Question: I'm using Polars to transform my DataFrame, and I want to chain multiple with_columns transformations. However, I encounter an issue when trying to perform operations on a newly created column within the same with_columns context. I end up needing to save the DataFrame after each transformation and then reapply with_columns for subsequent transformations. Is there a cleaner way to achieve this? Here is an example of my current approach: import polars as pl # Sample data exampledata = { 'A': [1, 2, 3], 'B': [4, 5, 6] } df = pl.DataFrame(exampledata) # First transformation df = df.with_columns( (pl.col("A") + pl.col("B")).alias("C") ) # Second transformation df = df.with_columns( (pl.col("C") * pl.col("B")).alias("D") ) print(df)  In this example, I create a new column C from columns A and B. Then, I need to save the DataFrame before I can create column D from C and B. Is there a more efficient or idiomatic way to chain these transformations in Polars?

Answer: For performance reasons, all expressions in a single pl.DataFrame.with_columns context are evaluated in parallel. Especially, it is not possible to use a column resulting from an expression in a different expression in the same context.1 1 This does not mean that polars performs duplicate work for expressions with common sub-expressions since under the hood a Common-Subplan-Elimination mechanism is used.  Still, since with_columns returns the resulting dataframe, you do not need to assign each result back to df, but can chain the with_columns calls directly as follows. df = ( df .with_columns( (pl.col("A") + pl.col("B")).alias("C") ) .with_columns( (pl.col("C") * pl.col("B")).alias("D") ) )  shape: (3, 4) ┌─────┬─────┬─────┬─────┐ │ A ┆ B ┆ C ┆ D │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╪═════╡ │ 1 ┆ 4 ┆ 5 ┆ 20 │ │ 2 ┆ 5 ┆ 7 ┆ 35 │ │ 3 ┆ 6 ┆ 9 ┆ 54 │ └─────┴─────┴─────┴─────┘
Question: I am working on a large project in VSCode. I have various subfolders with plenty of .py files, and a main.py in the project root. I want to import various functions from various files that exist in different subfolders. I find from x import y a very redundant process. What is the efficient way to do it? How does a professional python developer do it? my_project/ ├── FolderA/ │ ├── __init__.py │ └── module_a.py ├── FolderB/ │ ├── __init__.py │ └── module_b.py ├── FolderC/ │ ├── __init__.py │ └── module_c.py ├── __init__.py └── main.py  Let's say the above is my structure and I want to call functions in main.py from module_a.py, module_b.py and module_c.py. Thanks. I tried the following, but I can't keep doing it for 10/20 functions: from FolderA.module_a import functionA from FolderB.module_b import functionB from FolderC.module_c import functionC  Also tried the following, it didn't work: sys.path.append("D:\Python_workspace\software") 

Answer: From a practical standpoint (if the folder representation you gave is accurate to your project) if you find yourself with lots of folders with one file each in them, you may want to reassess whether the categories/groupings you've chosen are realistic - too granular and you get a lot of the repetitive, snakey imports that you've shown. Unfortunately at that point with one file per folder there's not much you can do to solve your import problem short of restructuring the project tree. However, if your project is already in the following format (difference being multiple files in each subfolder): my_project/ ├── FolderA/ │ ├── __init__.py │ ├── module_a1.py │ └── module_a2.py ├── FolderB/ │ ├── __init__.py │ ├── module_b1.py │ └── module_b2.py ├── FolderC/ │ ├── __init__.py │ ├── module_c1.py │ └── module_c2.py ├── __init__.py └── main.py  You can then set up imports in __init__.py (assume you're in FolderA/__init__.py): from module_a1 import fun1 from module_a2 import fun2  Then in any other file you can now just directly reference the directory/package and to grab what you've now indirectly imported: from FolderA import fun1, fun2
Question: I have a project in Django Rest Framework where I need have endpoint where I'm able to search Document objects by title and text and possibility to search object is active and inactive, using to this URL address. I am using to achieve this Django_filters package. Example: https://localhost:8000/?is_active=True. This is my model class Document(models.Model): title = models.CharField(max_length=100) text = models.TextField() date = models.DateField() account = models.ForeignKey( to=Account, null=True, blank=True, on_delete=models.CASCADE, related_name='documents', verbose_name='account', help_text='account', ) is_active = models.BooleanField(default=True) def __str__(self): return self.title  Serializer class DocumentSerializer(serializers.ModelSerializer): class Meta: model = Document fields = ['id', 'title', 'text', 'date', 'account', 'is_active']  Custom FilterSet class DocumentBackendFilter(dfilters.FilterSet): is_active = dfilters.BooleanFilter(field_name='is_active', lookup_expr='exact') class Meta: model = Document fields = ['is_active']  View class DocumentsListView(viewsets.ModelViewSet): queryset = Document.objects.all() serializer_class = DocumentSerializer filterset_class = DocumentBackendFilter filterset_fields = ('is_active',) filter_backends = [filters.SearchFilter, filters.OrderingFilter] search_fields = ['title', 'text'] ordering_fields = ['title', 'text'] def get_queryset(self): qs = super().get_queryset() qs = qs.filter(account__users=self.request.user) return qs  Problem: Under url: http://127.0.0.1:8000/?is_active=True I have elements where is_active is set on True and False. Default search is working OK. How can I get search, ordering and filter objects based on is_active?

Answer: since you want to use the DocumentBackendFilter, you must add the DjangoFilterBackend from django_filters.rest_framework to the filter_backends list in your view class. Besides, If you set filterset_class, the view class will ignore the value list of filterset_fields, so you should provide one of them. class DocumentsListView(viewsets.ModelViewSet): # some code filterset_class = DocumentBackendFilter # filterset_fields = ('is_active',) -> remove this line filter_backends = [filters.SearchFilter, filters.OrderingFilter, DjangoFilterBackend]
Question: I'm using popen() to create potentially long-running processes in Python. If the parent program dies and then restarts, is there a way to retrieve the previously created processes that are still running? I'd probably have to use start_new_session=True, which I'm not doing now. Essentially, I want to get a re-constructed instance of a popen object that points to the child process. I suspect it's not possible, since serializing a popen() object doesn't seem to be possible.

Answer: First of all, if one of the processes connected with a pipe dies, any read/write from/to that pipe will result in SIGPIPE -- which by default terminates the process. So your orphan process is very likely to die promptly, too, if the parent crashes.  As per Can I share a file descriptor to another process on linux or are they local to the process?, it's only possible to share a file discriptor between unrelated processes via a UNIX domain socket, and that requires active participation from both processes. As others have pointed out, you also won't be able to wait for that process.  Depending on what you're trying to achieve (e.g. it's a worker process and you don't want to lose its progress), a better way may be to e.g. save the progress at some points (whether by the parent or by the child) and add an ability to restart from a saved state.
Question: I have defined multiple functions where I search for a specific pydantic model in a list of pydantic models based on some attribute value. SocketIOUserSessionID = str RoomWithIndex = tuple[Room, RoomIndex] RSStateWithIndex = tuple[RSState, int] RSPacketSavedRecordWithIndex = tuple[RSPacketSavedRecordsContainer, int] def find_room_by_id(self, id: UUID | str, where: list[Room]) -> RoomWithIndex | None: room = next(filter(lambda room: room.id == id, where), None) if room is None: return None index = where.index(room) return room, index def find_room_by_session(self, session: SocketIOUserSessionID, where: list[Room]) -> RoomWithIndex | None: room = next(filter(lambda room: session in room.sessions, where), None) if room is None: return None index = where.index(room) return room, index def find_rs_state_by_room_id(self, room_id: str, where: list[RSState]) -> RSStateWithIndex | None: rs_state = next(filter(lambda rs_state: rs_state.room_id == room_id, where), None) if rs_state is None: return None index = where.index(rs_state) return rs_state, index def find_saved_record_by_room_id(self, room_id: str, where: list[RSPacketSavedRecordsContainer]) -> RSPacketSavedRecordWithIndex | None: saved_record = next(filter(lambda saved_records: saved_records.room_id == room_id, where), None) if saved_record is None: return None index = where.index(saved_record) return saved_record, index  How to write a generic function (with typing) to refactor such code? I heard of functools.singledispatch decorator but I am not sure that this is the right case to use it.  def find_value_by_attr(self): ? 

Answer: I tried to generalize the four functions as much as possible - here's where I ended up: def find_model( self, iid: UUID | str, where: list[Any], filter_attr: str, ) -> tuple[Any, int] | None: if ( model := next( filter(lambda model: iid == getattr(model, filter_attr), where) ) ): return model, where.index(model)  The find_model function takes  an iid (item ID) in place of id, session, or room_id (NOTE: naming this iid avoids shadowing the builtin id) a where list just as before a filter_attr which is the string name of the attribute you want to filter, e.g. 'id', 'sessions', or 'room_id'  The only thing this doesn't quite cover is the filtering case of find_room_by_session which is using in instead of == in the filter lambda. If anyone more clever than I would care to weigh in, I'm open to it! If I may take a moment to editorialize: I like the idea of using @singledispatch for this, but it doesn't get you away from writing multiple function prototypes anyway...if the goal it 'less code', then @singledispatch doesn't help much in that regard.
Question: I'm passing various bits of data to a function that computes variance along the first dimension. Sometimes the variance is zero, which is fine, but then the following strange thing happens: >> sigma = data.var(axis=0) + 1e-7 # data has zero variance so all entries should equal 1e-7 >> sigma array([1.e-07, 1.e-07, 1.e-07, ..., 1.e-07, 1.e-07, 1.e-07], dtype=float16) >> (sigma==1e-7).all() True >> sigma[0]==1e-7 False  On its own, the fourth line would be explained by the 16-bit precision, and indeed >> np.float16(1e-7)==1e-7 False  But it seems to contradict the third line, which says they are equal. This was causing a bug in my code. I can redesign around it, but I want to understand why numpy is doing this so I'm not caught out again in the future.

Answer: This comes from the fact that numpy type promotion treats scalars and arrays differently. You can see this with np.result_type: >>> np.result_type(sigma, 1E-7) dtype('float16') >>> np.result_type(sigma[0], 1E-7) dtype('float64')  Essentially, when an array value is compared to a scalar value (the first case), the dtype of the array value takes precedence. When comparing two scalars or two arrays (the second case), the highest precision takes precedence. What this means is that when you evaluate (sigma == 1E-7), both sides are first cast to float16 before comparison, whereas when you evaluate sigma[0] == 1E-7, both sides are first cast to float64 before comparison. Because float16 cannot perfectly represent the value 1E-7, this causes a discrepency in the scalar-comparison case, where both values are cast to float64: >>> np.float16(1E-7).astype(np.float64) 1.1920928955078125e-07 >>> np.float64(1E-7) 1e-07  Finally, please note that these scalar-specific type casting rules are being changed in NumPy 2.0 (see NEP 50: Promotion Rules for Scalars), so if you run your code with NumPy 2.0, both cases will promote to float16 and return True.
Question: I have made a subplot in matplotlib and managed to to put the different cmap I have on the same column. For a minimal working example (with dummy cmaps): import matplotlib.pyplot as plt import numpy as np # Generate sample data data1 = np.random.rand(10, 10) data2 = np.random.rand(10, 10) data3 = np.random.rand(10, 10) fig_bandwidth = plt.figure(figsize=(12, 6)) ax1 = plt.subplot(3, 2, 6) ax2 = plt.subplot(3, 2, 4) ax3 = plt.subplot(3, 2, 2) ax_bandwidth = plt.subplot(1, 3, 1) axes = [ax1, ax2, ax3] # Plot data and add color bars for ax, data in zip(axes, [data1, data2, data3]): cax = ax_bandwidth.imshow(data, aspect='auto', cmap='viridis') plt.colorbar(cax, ax=ax) ax.axis('off') plt.tight_layout() plt.show()   What I am trying to do is have a tight subplot with the figure on the left and the 3 color bars on the right in the same column, but it seems the plotting boxes are still there, preventing me from placing these axes next to the figure. Maybe using subplots isn't the best solution, any suggestion? Then how could I place an ax title spanning across the three color bars since they represent the same thing (bandwidth in MHz for context).

Answer: Using add_axes works much better than sublots. It gives me much more freedom on the placement. Here is a minimal working example with dummy cbars: fig_bandwidth = plt.figure(figsize=(12, 6)) # Creating three axes: add_axes([xmin,ymin,dx,dy]) ax1 = fig_bandwidth.add_axes((0.75, 0.05, 0.1, 0.3)) ax2 = fig_bandwidth.add_axes((0.75, 0.36, 0.1, 0.3)) ax3 = fig_bandwidth.add_axes((0.75, 0.67, 0.1, 0.3)) ax_bandwidth = fig_bandwidth.add_axes((0.1, 0.05, 0.7, 0.92)) axes = [ax1, ax2, ax3] # Generate sample data data1 = np.random.rand(10, 10) data2 = np.random.rand(10, 10) data3 = np.random.rand(10, 10) for ax, data in zip(axes, [data1, data2, data3]): cax = ax_bandwidth.imshow(data, aspect='auto', cmap='viridis') plt.colorbar(cax, ax=ax) ax.axis('off') plt.show()