eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I was experimenting with refcount of objects, and I noticed the reference count of None object does not change when I bind identifiers to None. I observed this behavior in python version 3.13. Python 3.13.0 (main, Oct 7 2024, 05:02:14) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.getrefcount(None) 4294967295 >>> list_of_nones = [None for _ in range(100)] >>> sys.getrefcount(None) 4294967295 >>> del list_of_nones >>> sys.getrefcount(None) 4294967295  This behavior is in contrast with the behavior of python 3.10: Python 3.10.15 (main, Sep 7 2024, 00:20:06) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> >>> sys.getrefcount(None) 4892 >>> list_of_nones = [None for _ in range(100)] >>> sys.getrefcount(None) 4990 >>> >>> del list_of_nones >>> >>> sys.getrefcount(None) 4890  In the 3.10 version, the reference count of None, decrease and increase according to binding identifiers and deleting them. But in the 3.13, the reference count is always fixed. Can someone explain this behavior?

Answer: This is due to PEP 683. To avoid any need to ever write to the memory of certain "immortal" objects, like None, those objects now have a fixed refcount that never changes, no matter how many actual references to the object exist. This helps with multi-threaded and multi-process performance, avoiding issues with things like cache invalidation and copy-on-write.
Question: I have 2 dataframes that look like this: import pandas as pd data = {'QuarterYear': ["Q3 2023", "Q4 2023", "Q1 2024", 'Q2 2024', "Q3 2024", "Q4 2024"], 'data1': [5, 6, 2, 1, 10, 3], 'data2': [12, 4, 2, 7, 2, 9], 'data3': [2, 42, 2, 6, 2, 4]} df = pd.DataFrame(data) This looks like: QuarterYear data1 data2 data3 0 Q3 2023 5 12 2 1 Q4 2023 6 4 42 2 Q1 2024 2 2 2 3 Q2 2024 1 7 6 4 Q3 2024 10 2 2 5 Q4 2024 3 9 4 data1 = {'QuarterYear': ["Q4 2023", 'Q2 2024', "Q3 2024"], 'data1': [5, 9, 10], 'data2': [7, 7, 3], 'data3': [2, 11, 3]} df1 = pd.DataFrame(data1) This looks like: QuarterYear data1 data2 data3 0 Q4 2023 5 7 2 1 Q2 2024 9 7 11 2 Q3 2024 10 3 3  What I would like to do is get df1 and make it the same size as df1, that is fill in for the other quarters with values for data1,2, and 3 being 0 if they are not already in df1. So the end result should look like this:  QuarterYear data1 data2 data3 0 Q3 2023 0 0 0 1 Q4 2023 5 7 2 2 Q1 2024 0 0 0 3 Q2 2024 9 7 11 4 Q3 2024 10 3 3 5 Q4 2024 0 0 0 

Answer: Use DataFrame.set_index with DataFrame.reindex: out = (df1.set_index('QuarterYear') .reindex(df['QuarterYear'], fill_value=0) .reset_index()) print (out) QuarterYear data1 data2 data3 0 Q3 2023 0 0 0 1 Q4 2023 5 7 2 2 Q1 2024 0 0 0 3 Q2 2024 9 7 11 4 Q3 2024 10 3 3 5 Q4 2024 0 0 0  Another idea: out = df1.merge(df[['QuarterYear']], how='right').fillna(0) print (out) QuarterYear data1 data2 data3 0 Q3 2023 0.0 0.0 0.0 1 Q4 2023 5.0 7.0 2.0 2 Q1 2024 0.0 0.0 0.0 3 Q2 2024 9.0 7.0 11.0 4 Q3 2024 10.0 3.0 3.0 5 Q4 2024 0.0 0.0 0.0  Thank you @ouroboros1 for comment:  In both cases, you're assuming that df['QuarterYear'] at least contains all values in df1['QuarterYear']. In OP's example, that the case, but if it were not, both methods will actually filter out rows from df1. That does not seem to be what the OP wants. E.g. change "Q3 2024" to "Q3 2025" for df, and "Q3 2024" will be gone from the result. So, with reindex, you would in that case need a union, with merge outer.  out = df1.merge(df[['QuarterYear']], how='outer').fillna(0) print (out) QuarterYear data1 data2 data3 0 Q1 2024 0.0 0.0 0.0 1 Q2 2024 9.0 7.0 11.0 2 Q3 2023 0.0 0.0 0.0 3 Q3 2024 0.0 0.0 0.0 4 Q3 2025 10.0 3.0 3.0 5 Q4 2023 5.0 7.0 2.0 6 Q4 2024 0.0 0.0 0.0 out = (df1.set_index('QuarterYear') .reindex(pd.Index(df['QuarterYear']).union(df1['QuarterYear']), fill_value=0) .reset_index()) print (out) QuarterYear data1 data2 data3 0 Q1 2024 0 0 0 1 Q2 2024 9 7 11 2 Q3 2023 0 0 0 3 Q3 2024 0 0 0 4 Q3 2025 10 3 3 5 Q4 2023 5 7 2 6 Q4 2024 0 0 0  If necessary, here is solution also for sorting by quarters: out = (df1.set_index('QuarterYear') .reindex(pd.Index(df['QuarterYear']).union(df1['QuarterYear']), fill_value=0) .reset_index() .sort_values('QuarterYear', key=lambda x: pd.to_datetime(x.str[-4:] + x.str[:2], format='mixed'), ignore_index=True)) print (out) QuarterYear data1 data2 data3 0 Q3 2023 0 0 0 1 Q4 2023 5 7 2 2 Q1 2024 0 0 0 3 Q2 2024 9 7 11 4 Q3 2024 0 0 0 5 Q4 2024 0 0 0 6 Q3 2025 10 3 3
Question: I want to write the below type of query in Python. But basic python filtering acts like I did use WHERE clause in SQL, not ON for filtering. Could anyone please help? Appreciate for your support. select * from t1 left join t2 on t1.key = t2.key and t2.x2 <= t1.x and t2.y2 > t1.y  I tried the below Python code, and it is not working same with SQL query. df = t1.merge(t2, how = 'left', on = 'key') df = df[df[x2] <= df[x]] df = df[df[y2] > df[y]] 

Answer: Merge the dataframes, then check which locs does not match the condition, then set those locs as NaN. While converting to NaN the datatype becomes float, so you might need to convert them to integer. t1_data = {'key': [1, 2, 3, 4], 'x': [5, 6, 7, 8], 'y': [10, 11, 12, 13]} t2_data = {'key': [2, 3, 4], 'x2': [6, 7, 8], 'y2': [16, 17, 18]} t1 = pd.DataFrame(t1_data) t2 = pd.DataFrame(t2_data) merged = pd.merge(t1, t2, on='key', how='left') condition = (merged['x2'] <= merged['x']) & (merged['y2'] > merged['y']) # This identifies the rows which does not satisfy the condition, and then set x2 and y2 to NaN merged.loc[~condition & merged['y2'].notna(), ['x2', 'y2']] = pd.NA  Pandas Output  key x y x2 y2 0 1 5 10 NaN NaN 1 2 6 11 6.0 16.0 2 3 7 12 7.0 17.0 3 4 8 13 8.0 18.0  The output with SQL matches with the Pandas output. Fiddle SQL Output key x y x2 y2 1 5 10 null null 2 6 11 6 16 3 7 12 7 17 4 8 13 8 18
Question: I am working with django-tables2 to display some patient information on a page. I am creating the table like this: class PatientListView(tables.Table): name = tables.Column('Practice') patientid = tables.Column() firstname = tables.Column() lastname = tables.Column() dob = tables.Column() addressline1 = tables.Column() addressline2 = tables.Column() city = tables.Column() state = tables.Column() zipcode = tables.Column() class Meta: template_name = 'django_tables2/bootstrap.html'  and then I am populating the table in my view with the result of an sql query like this: table = PatientListView(patients)  I would like to ideally make each row of the table clickable so clicking anywhere on the table row would take me to a separate url defined by me. I would also settle for having a specific cell to click that would take me to a separate url. I have seen the linkify option, but from what I've read of the documentation it looks like linkify does redirects to django model pages, but I am not using models for this database as the database is created and managed by another application, and I am just reading and displaying that information. If django-tables2 is not the right solution for this issue I am open to hearing suggestions of other ways I can accomplish my goal.

Answer: Option 1: turn every column into a link You can make a callable that converts the record to the link, and add that to all columns, so: def get_link(record): return f'www.example.com/patients/{record.patientid}' class PatientListView(tables.Table): name = tables.Column('Practice', linkify=get_link) patientid = tables.Column(linkify=get_link) firstname = tables.Column(linkify=get_link) lastname = tables.Column(linkify=get_link) dob = tables.Column(linkify=get_link) addressline1 = tables.Column(linkify=get_link) addressline2 = tables.Column(linkify=get_link) city = tables.Column(linkify=get_link) state = tables.Column(linkify=get_link) zipcode = tables.Column(linkify=get_link) class Meta: template_name = 'django_tables2/bootstrap.html'  Option 2: make the row clickable Another option is to generate a data-href attribute and use JavaScript then to make it behave like a link, with: def get_link(record): return f'www.example.com/patients/{record.patientid}' class PatientListView(tables.Table): name = tables.Column('Practice') patientid = tables.Column() firstname = tables.Column() lastname = tables.Column() dob = tables.Column() addressline1 = tables.Column() addressline2 = tables.Column() city = tables.Column() state = tables.Column() zipcode = tables.Column() class Meta: row_attrs = {'data-href': get_link}  and then add some JavaScript:  <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"> </script> $(function() { $('tr[data-href]').on('click', function() { window.location = $(this).data('href'); }); });  and perhaps style the row with:  tr[data-href] { cursor: pointer; }
Question: I've tried with 3 different parsers: lxml, html5lib, html.parser All of them output invalid HTML: >>> BeautifulSoup('<br>', 'html.parser') <br/> >>> BeautifulSoup('<br>', 'lxml') <html><body><br/></body></html> >>> BeautifulSoup('<br>', 'html5lib') <html><head></head><body><br/></body></html> >>> BeautifulSoup('<br>', 'html.parser').prettify() '<br/>\n'  All of them have /> "self-closing" void tags. How can I get BeautifulSoup to output HTML that has void tags without />?

Answer: Use the html5 formatter:  If you pass in formatter="html5", it’s the same as formatter="html", but Beautiful Soup will omit the closing slash in HTML void tags like “br”:  from bs4 import BeautifulSoup BeautifulSoup('<br>', 'html.parser').decode(formatter="html5")  Which outputs: '<br>'
Question: I am attempting to create a nested dictionary 'world'. Due to the planned size and complexity I am hoping to automate the creation a bit. However, when I attempt to run it, I get "AttributeError: 'Pathway' object has no attribute 'hidden'". The intended structure of the dictionary is as follows **world  village a. rooms i. villageEntrance 1. hidden : false 2. pathways: [p1,etc] ii. etc b. alignment : good c. etc void a. rooms i. voidRoom 1. hidden : false 2. pathways: [p1] b. alignment : neutral c. etc**  class Pathway(dict): def __init__(self, hidden, travelType): self.dict = { self.hidden : hidden, self.travelType : travelType } class Room(dict): def __init__(self, hidden, pathways): self.dict = { self.hidden : hidden, self.pathways : pathways } class Location(dict): def __init__(self, rooms, alignment): self.dict = { self.alignment : alignment, self.rooms : rooms } p1 = Pathway(True, "teleport") voidRoom = Room(True, [p1]) villageEntrance = Room(False, [p1]) void = Location([voidRoom], "neutral") village = Location([villageEntrance], "good") world = { "void" : void, "village" : village } def locationDataRetrieval(path): val = world.get(path) return val print(world)  I looked up proper syntax for dictionaries and as far as I can see I have it correct. I also attempted removing the pathway class but the error moved on to the next class.

Answer: This is python, not javascript. class Pathway(dict): def __init__(self, hidden, travelType): self.dict = { self.hidden : hidden, self.travelType : travelType }  PEP 8 asks you to instead name it travel_type. It's not strictly necessary to define all attributes in the __init__() constructor, but it's good practice, it's considered polite. Here, you're attempting to dereference self.hidden before you have assigned anything to it. A typical ctor would look like:  def __init__(self, hidden, travel_type): self.hidden = hidden self.travel_type = travel_type  After that it's perfectly fine to dereference self.hidden. You can even put it in a dict if you like. It's unclear why that would be desirable, though. I recommend you settle on object attributes, or settle on dicts, without trying to mix them. Storing the same thing in two places tends to lead to maintenance trouble later on.  A @dataclass decorator can save you the trouble of writing a ctor: from dataclasses import dataclass @dataclass class Pathway: hidden: bool travel_type: str
Question: SETUP I have the following df: import pandas as pd import numpy as np arrays = [ np.array(["fruit", "fruit", "fruit","vegetable", "vegetable", "vegetable"]), np.array(["one", "two", "total", "one", "two", "total"]), ] df = pd.DataFrame(np.random.randn(6, 4), index=arrays) df.index.set_names(['item','count'],inplace=True)  WHAT I AM TRYING TO DO I am trying to style df so that each cell where count == 'total' is bolded. WHAT I HAVE TRIED I was able to index all rows where count == 'total' with the following code: idx = pd.IndexSlice totals = df.loc[idx[:, 'total'],:]  but when I try to apply a function: def df_style(val): return "font-weight: bold" df.style.applymap(df_style,subset=totals)  I get the following error: KeyError: 0   How can I style this df so that all cells where count == 'total' are bolded? Here is a similar question, albeit with just a regular index rather than MultiIndex.

Answer: Here's one approach: # used `np.random.seed(0)` for reproducibility def highlight_total(s): m = s.index.get_level_values('count') == 'total' return np.where(m, 'font-weight: bold', None) df.style.apply(highlight_total)  Output:  Explanation  For level 'count' (or: '1'), check where index.get_level_values equals 'total'. Pass the result to np.where. Use inside Styler.apply.   Specifically traversing the df row-wise, you could do: def highlight_total(s): return ['font-weight: bold']*len(s) if s.name[1] == 'total' else [None]*len(s) df.style.apply(highlight_total, axis=1)  So, here we are accessing s.name[1], with name understood as ('fruit', 'total'), ('vegetable', 'total'), etc. Leads to the same result.
Question: How can I keep the order of my stacked bars chart from my Dataframe ? The head of my Dataframe looks like this :  The countries are ordered as I want them to be and I can handle it by setting sort=None. But I want to order lineages in the stacked bar by sequences_number, only keeping the 'Others' value at the end, as it is in my Dataframe. By using sort=None in the X encoding channel, it still sort lineages alphabetically. So I added a 'rank' column in my Dataframe in order to use the sort=alt.SortField("rank:Q", order="ascending") but it still order lineage alphabetically instead of by sequences_number (with 'Others' at the end). Here is my code : histo_base_proportion = ( alt.Chart( top10_selection_africa ).encode( alt.Y( "country:N", scale=alt.Scale(padding=0.3), sort=None #sorting works here ), alt.X( "sequences_number:Q", sort=alt.SortField("rank:Q", order="ascending"), #but does not work here title = "Lineages", stack="normalize" ), alt.Color( "lineage" ).scale( domain=fixed_domain_lineages, range=fixed_range_lineages ).legend(None), tooltip=[ {"type": "nominal", 'title': 'Country',"field": "country"}, {"type": "nominal", 'title': 'Lineage',"field": "lineage"}, {"type": "quantitative", 'title': 'Nombres', "field": "sequences_number"} ] ).mark_bar( size=12 ).properties( width=800, title="Proportion of sequenced genomes lineage by country in " + year_title ) )  Here is what I get :  This represents the lineages number proportion by country. Top 5 lineages is shown per country, lineages outside of top 5 are in 'Others' category, in black color. In my case, I would have wanted the first green lineage of Rwanda to be just before 'Others' (black color) or the dark blue lineage of Comoros to be just before 'Others' (black color), for examples. How can I do that ? Thanks in advance.

Answer: You can use the order encoding instead of sort to order the stacked segments as in this example: import altair as alt from vega_datasets import data source = data.barley() alt.Chart(source).mark_bar().encode( x='sum(yield)', y='variety', color='site', order=alt.Order( # Sort the segments of the bars by this field 'site', sort='ascending' ) )  You can read more about how order works in the documentation
Question: I use Pyspark in Azure Databricks to transform data before sending it to a sink. In this sink any array must at most have a length of 100. In my data I have an array that is always length 300 an a field specifying how many values of these are relevant (n_relevant). n_relevant values might be:  below 100 -> then I want to keep all values between 100 and 300 -> then I want to subsample based on modulo above 300 -> then I want to subsample modulo 3  E.g.: array: [1,2,3,4,5,...300] n_relevant: 4 desired outcome: [1,2,3,4] array: [1,2,3,4,5,...300] n_relevant: 200 desired outcome: [1,3,5,...199] array: [1,2,3,4,5,...300] n_relevant: 300 desired outcome: [1,4,7,...298] array: [1,2,3,4,5,...300] n_relevant: 800 desired outcome: [1,4,7,...298]  This little program reflects the desired behavior: from math import ceil def subsample(array:list,n_relevant:int)->list: if n_relevant<100: return [x for i,x in enumerate(array) if i<n_relevant] if 100<=n_relevant<300: mod=ceil(n_relevant/100) return [x for i,x in enumerate(array) if i%mod==0 and i<n_relevant] else: return [x for i,x in enumerate(array) if i%3==0] n_relevant=<choose n> t1=[i for i in range(300)] subsample(t1,n_relevant)  What I have tried: transforms to set undesired values to 0 and remove those with array_remove could subset with a specific modulo BUT cannot adopt to n_relevant. Specifically you cannot hand a parameter to the lambda function and you cannot dynamically change the function.

Answer: You can filter by index as follows from pyspark.sql.types import StructField, StructType, IntegerType, ArrayType df = spark.createDataFrame( [[list(range(300)), 4], [list(range(300)), 200], [list(range(300)), 300], [list(range(300)), 800]], schema=StructType( [ StructField("array", ArrayType(IntegerType())), StructField("n_relevant", IntegerType()), ] ), ) df = df.withColumn( "result", F.when(F.col("n_relevant") <= 100, F.slice("array", 1, F.col("n_relevant"))) .when( F.col("n_relevant") <= 200, F.filter( F.slice("array", 1, F.col("n_relevant")), lambda _, index: index % 2 == 0 ), ) .otherwise( F.filter( F.slice("array", 1, F.col("n_relevant")), lambda elem, index: index % 3 == 0 ) ), ) display(df)
Question: I have a grayscale camera for which I have already calculated intrinsic parameters with standard methods of calibrations. I have then position this camera in a particular stationary setup and put a plate with 8 marker points in front of the camera. I have calculated the camera pose with respect to the coordinate system of those markers using the formula: cameraPosition = -np.matrix(rotM).T * np.matrix(tvec), where rotM is the rotational matrix and tvec is the translation vector obtained with opencv's cv.solvePnP I wanted to check, how accurate the calculated camera pose is, but have stumbled into a problem, which is, that I do not know, to what physical point does the cameraPosition vector points to. I firstly thought that it points to the center of the image plane, but now I've read that it points to the center of projection point. Which is it? And if it's indeed the center of projection, how could I calculate, where that is located in my camera (using a certain camera settings)? Thanks

Answer: -np.matrix(rotM).T * np.matrix(tvec) is the reverse operation from converting a point (xw yw zw) in the world coordinates into camera coordinates, when that point is (0,0,0) in camera system. [R R R tx][xw] [R R R ty][yw] [R R R tz][zw] [0 0 0 1 ][1 ]  is Xc = rotM@Xw + tvec (Xw=[xw,yw,zw], tvec=[tx,ty,tz], rotM=R) So reverse operation is Xw = RotM⁻¹@(Xc-tvec) aka Xw = RotM.T@(Xc-tvec), since, for a rotation matrix, inverse and transpose are the same. If Xc=[0,0,0], that is Xw=-rotM.T@tvec So, indeed, that points to the center of projection (origin of camera coordinates system), expressed in the world coordinates. It tells you where the camera is in your world coordinates (the one you used to position physical points in the 3D world) To end up in the center of the image, that takes a projection (canonical projection matrix, times intrisinc matrix), that would lead to [cx,cy], the position of the camera direction in the image (usually the middle of the image, but not necessarily). But the only point in the world that you cannot project happens to be the center of the projection. So to get that, you need first to add something (anything) to Zc So, I do not know how to you intend to use that "center of projection expressed in world coordinates" to check your projection. Maybe to verify that, in your world coordinates, that point indeed to where the physical camera is? (in reality, not exactly: that is the center of projection. It is a virtual point created by lens. But, well, it should be not far from a point "inside" the camera, at a focal distance from the lens.
Question: I ran below to get information about the list of arguments and default values of a function/method. import pandas as pd import inspect inspect.getfullargspec(pd.drop_duplicates)  Results: Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'pandas' has no attribute 'drop_duplicates' >>> inspect.getfullargspec(pd.drop_duplicates)  What is the correct way to fetch such information from any function/method?

Answer: The issue is that the drop_duplicates method exists as a method on a pandas.DataFrame object, i.e., pandas.DataFrame.drop_duplicates not directly on the pandas module. With that being said you might also want to check out inspect.signature as an alternative to inspect.getfullargspec: >>> import inspect >>> import pandas as pd >>> method = pd.DataFrame.drop_duplicates >>> inspect.getfullargspec(method) FullArgSpec(args=['self', 'subset'], varargs=None, varkw=None, defaults=(None,), kwonlyargs=['keep', 'inplace', 'ignore_index'], kwonlydefaults={'keep': 'first', 'inplace': False, 'ignore_index': False}, annotations={'return': 'DataFrame | None', 'subset': 'Hashable | Sequence[Hashable] | None', 'keep': 'DropKeep', 'inplace': 'bool', 'ignore_index': 'bool'}) >>> inspect.signature(method) <Signature (self, subset: 'Hashable | Sequence[Hashable] | None' = None, *, keep: 'DropKeep' = 'first', inplace: 'bool' = False, ignore_index: 'bool' = False) -> 'DataFrame | None'>
Question: I am trying to make a table of contents for my queryset in Django like this: def get_toc(self): toc = {} qs = self.get_queryset() idx = set() for q in qs: idx.add(q.title[0]) idx = list(idx) idx.sort() for i in idx: toc[i] = [] for q in qs: if q.title[0] == i: toc[i].append(q) return toc  But it has time complexity O(n^2). Is there a better way to do it? UPDATE I meant glossary, not table of contents.

Answer: This doesn't look like a table of contents, but a glossary, where you map the first character of a term to a list of terms. We can work with .groupby(…) [python-doc] here: from itertools import groupby result = { k: list(vs) for k, vs in groupby( self.get_queryset().order_by('title'), lambda x: x.title[0] ) }
Question: I'm trying to set up authentication for my OPC-UA server. I don't want my clients to be able to connect to my server in ‘anonymous’ mode. So I used this configuration in my opc-ua server:  I'm testing with UaExpert in client mode and the password login works (I have to enter the right login + the right password). The certificate login is blocked, which is fair enough. However, I can still connect in ‘anonymous’ mode, which obviously doesn't work for me. The logs from UaExpert when connecting in "anonymous":  I'd prefer to use the certificate authentication mode but I wanted to test the password login first. I've tested the certificate mode and I can also connect in anonymous mode! Does anyone have any ideas?

Answer: The python opcua library is no longer supported. There was a fix for this issue, but the pip package never got uppdated. So either use the current master from github. Or you switch to asyncua, which has a sync layer for easier porting, but i would recommend use it via async, if possible.