eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Campaign Record { "_id": { "$oid": "64eb81337f7d9f6e1107fc3a" }, "_cls": "CampaignModel", "campaignNumber": "CMPN-00005", "account": { "$oid": "64d80694fe75052b39c0e615" }, "campaignName": "title", "campaignDescription": "wdesc", "campaignStartDate": { "$date": "2023-08-17T00:00:00.000Z" }, "campaignEndDate": { "$date": "2023-08-27T00:00:00.000Z" }, "isWorkOrderGeneratedFlag": false, "isInvoiceGenerated": false, "campaignCost": "20000", "clientName": "", "attachedBillboardList": [], "accountId": "64d80694fe75052b39c0e615" }  Booking Record { "_id": { "$oid": "64eb81337f7d9f6e1107fc3b" }, "_cls": "BookingModel", "campaignId": "64eb81337f7d9f6e1107fc3a", "billboardId": "64e0d18aaec5d7ef7bf848dd", "startDate": { "$date": "2023-09-01T00:00:00.000Z" }, "endDate": { "$date": "2023-09-10T00:00:00.000Z" }, "costPerDay": 200, "billboard": { "_ref": { "$ref": "BillBoard", "$id": { "$oid": "64e0d18aaec5d7ef7bf848dd" } }, "_cls": "BillBoardModel" } }  I am trying to join the campaign with booking on _id from the campaign record and campaignId in the booking record. Campaign Mongoengine class class CampaignModel(BaseModel): meta = {'collection': 'Campaign'} campaignNumber = me.SequenceField(sequence_name="CAMPAIGN", value_decorator=generate_campaign_number) account = me.ReferenceField(UserModel) campaignName = me.StringField() campaignDescription = me.StringField() campaignStartDate = me.DateField(default=datetime.datetime.now) campaignEndDate = me.DateField(default=datetime.datetime.now) isWorkOrderGeneratedFlag = me.BooleanField(default=False) isInvoiceGenerated = me.BooleanField(default=False) campaignCost = me.StringField() clientName = me.StringField() attachedBillboardList = me.ListField()  Booking Mongoengine Class class BookingModel(BaseModel): meta = {'collection': 'Booking'} campaignId = me.StringField() campaign = me.LazyReferenceField(CampaignModel,dbref=False) billboardId = me.StringField() accountId = me.StringField() billBoard = me.LazyReferenceField(BillBoardModel,dbref=False) startDate = me.DateField() endDate = me.DateField() costPerDay = me.FloatField()  query records = CampaignModel.objects.aggregate(*[ { "$lookup": { "from": "Booking", # Tag collection database name "localField": "_id", # Reference field "foreignField": "campaignId", # Primary key of the Tag collection "as": "attaches" } }])  Expected: Records from campaign collection with records from Booking collection as an array in attaches key. But the attaches key was empty.

Answer: The problem was you were storing the campaignId as a string type in the Booking collection. Make sure that both values to be compared must be same type before comparing. Thus, you need to convert the campaignId for the document in the Booking collection as ObjectId type as below: db.Campaign.aggregate([ { "$lookup": { "from": "Booking", let: { campaignId: "$_id" }, pipeline: [ { $match: { $expr: { $eq: [ "$$campaignId", { $toObjectId: "$campaignId" } ] } } } ], "as": "attaches" } } ])  Demo @ Mongo Playground
Question: In databricks I can set a config variable at session level, but it is not found in the context variables: spark.conf.set(f"dataset.bookstore", '123') #dataset_bookstore spark.conf.get(f"dataset.bookstore")#123 scf = spark.sparkContext.getConf() allc = scf.getAll() scf.contains(f"dataset.bookstore") # False  I understand there is a difference between session and context-level config variables, how can I retrieve all session-level variables using spark.conf? Note: all_session_vars = spark.conf.getAll() returns AttributeError: 'RuntimeConfig' object has no attribute 'getAll'  so it looks like a runtime-level config

Answer: Pyspark's RuntimeConfig indeed seems to be limited compared to its Scala counterpart. However if we peek at its source code... class RuntimeConfig: def __init__(self, jconf: JavaObject) -> None: """Create a new RuntimeConfig that wraps the underlying JVM object.""" self._jconf = jconf  we will be able to retrieve its underlying JVM object and through that call its getAll method: c = spark.conf._jconf.getAll() c.contains("dataset.bookstore") # True
Question: There is the PhoneNumber type to validate phone numbers: from pydantic import BaseModel from pydantic_extra_types.phone_numbers import PhoneNumber class User(BaseModel): name: str phone_number: PhoneNumber user = User(name='John', phone_number='+447911123456') print(user.phone_number) #> tel:+44-7911-123456  I'm satisfied with it except the unwanted "tel:" prefix. I applied a trick like that: from pydantic import BaseModel from pydantic_extra_types.phone_numbers import PhoneNumber class User(BaseModel): name: str phone_number: PhoneNumber @model_validator(mode="after") def remove_prefix_from_phone(self) -> "User": if self.phone_number: self.phone_number = self.phone_number.removeprefix("tel:") return self user = User(name='John', phone_number='+447911123456') print(user.phone_number) #> +44-7911-123456  But problem is that I convert the field into str and the PhoneNumber class is not even considered as str subtype unlike EmailStr for example. Seems like it a bad move and MyPy marks it as error as well - incompatible types in assignment (meanwhile, the validation logic is fine). Maybe I could do it smarter? I couldn't think of other options but it should be done within the Pydantic model for sure, because I load data from DB in a web-service and prefix would be exist in auto-generated response. Would be glad for any piece of advice!

Answer: The PhoneNumber class has these class variables: #https://github.com/pydantic/pydantic-extra-types/blob/092251d226edcf4e06bbe4f904da177fad20a6de/pydantic_extra_types/phone_numbers.py#L26 default_region_code: str | None = None phone_format: str = 'RFC3966' min_length: int = 7 max_length: int = 64  If we follow a bunch of code chains, we end up in the phonenumber repository that PhoneNumber depends on. This is the last stop for your data before being returned as a PhoneNumber, We can see that phone_format ultimately affects the phone number in the following ways: #https://github.com/daviddrysdale/python-phonenumbers/blob/2f06ef6db2ca83f3856fbb8019a0c665f5971b13/python/phonenumbers/phonenumberutil.py#L1726 def _prefix_number_with_country_calling_code(country_code, num_format, formatted_number): """A helper function that is used by format_number and format_by_pattern.""" if num_format == PhoneNumberFormat.E164: return _PLUS_SIGN + unicod(country_code) + formatted_number elif num_format == PhoneNumberFormat.INTERNATIONAL: return _PLUS_SIGN + unicod(country_code) + U_SPACE + formatted_number elif num_format == PhoneNumberFormat.RFC3966: #_RF3966_PREFIX = 'tel:' return _RFC3966_PREFIX + _PLUS_SIGN + unicod(country_code) + U_DASH + formatted_number else: return formatted_number  It may be notable that the above returns only the formatted number if you use an unrecognized format. There is a 'NATIONAL' format that is oddly absent from the above conditions. Using it should trigger the else. This should solve your problem. from pydantic import BaseModel from pydantic_extra_types.phone_numbers import PhoneNumber PhoneNumber.phone_format = 'E164' #'INTERNATIONAL', 'NATIONAL' class User(BaseModel): name: str phone_number: PhoneNumber
Question: How do I randomly sample 2d points uniformly from within an octagon using Python / Numpy? We can say that the octagon is centered at the origin (0, 0). The following is what I've done: import numpy as np import matplotlib.pyplot as plt def sample_within_octagon(num_points): points = np.zeros((num_points, 2)) # Generate random angle in radians angles = np.random.uniform(0, 2 * np.pi, size=(num_points,)) # Calculate the maximum radius for the given angle # This is wrong. max_radii = 1.0 / np.sqrt(2) / np.cos(np.pi / 8 - angles % (np.pi / 4)) # Generate random radius within the bounds of the octagon # Use square-root to prevent it from being more dense in center. radii = np.sqrt(np.random.uniform(0, max_radii)) # Convert polar coordinates to Cartesian coordinates x = radii * np.cos(angles) y = radii * np.sin(angles) points[:, 0] = x points[:, 1] = y return points num_points = 10000 random_points = sample_within_octagon(num_points) plt.scatter( np.array(random_points)[:, 0], np.array(random_points)[:, 1], s=1); plt.axis('equal');  The above code is mostly correct, but the max_radii calculation is incorrect, because the edges are slightly curved outward.  I am not necessarily committed to the overall approach of the above algorithm, so any algorithm will do. Having said that, I would slightly prefer an approach that (like the above, if it had actually worked correctly) would generalize to 16-gons and so on.

Answer: In your code, the formula for max_radii needs a little modification, try the following: import matplotlib.pyplot as plt import numpy as np from scipy import interpolate def sample_within_octagon(num_points, inv_transform_evals=10000): points = np.zeros((num_points, 2)) # Angle offset for each octagon segment offset = np.pi / 8.0 # Generate random angle in radians max_radii_in = np.linspace(0, 2 * np.pi, inv_transform_evals) max_radii_out = 1 / np.cos(np.abs(max_radii_in % (np.pi / 4) - offset)) max_radii_cdf = np.cumsum(max_radii_out / max_radii_out.sum()) f = interpolate.interp1d(np.array([0.] + list(max_radii_cdf)), np.array([0.] + list(max_radii_in))) angles_out = np.random.uniform(0, 1, num_points) angles = f(angles_out) # Calculate max radius based on octagon geometry max_radii = 1 / np.cos(np.abs(angles % (np.pi / 4) - offset)) # Generate random radius with square root scaling radii = np.sqrt(np.random.uniform(0, 1, num_points)) * max_radii # Convert to Cartesian coordinates points[:, 0] = radii * np.cos(angles) points[:, 1] = radii * np.sin(angles) return points # Generate and plot points num_points = 10000 points = sample_within_octagon(num_points) plt.scatter(points[:, 0], points[:, 1], s=1) plt.axis('equal') plt.show()  Note: The above solution has been modified by the OP - @calmcc based on suggestions in the comments of the question.
Question: I'm trying to publish a module to PyPi and when I run the command twine upload dist/* everything works fine, I can type in my name, but then I cannot type anything in the password field, other than clicking enter, I've tried googling it but found none of the solutions worked, there isn't much information on this anyway, help will be appreciated.

Answer: Um, this might be a really basic answer here (I can't comment on posts yet), but, are you sure you're not just dealing with the fact that by default in most command line interfaces/shells, you won't see any kind of "activity" (not even censored/starred out characters) when typing in a password? So yeah, you should just try typing out your password and then hitting enter. It can be a bit annoying when, say, connecting to a server via SSH and you have a lengthy passphrase to type on your private key, but think you might've missed a character (and know which one and where), but because it doesn't display anything, you can't use the arrow keys to go back and position the cursor to the right place to insert it. You have to just hit backspace an appropriate number of times and start over.
Question: I have some Python code that makes a HTTP request: import requests response = requests.get( url, cert = tuple(clientCertPath, pkeyPath), // paths to crt.pem and pkey.pem verify = serverCertPath // path to server-ca.crt file )  I'd like to rewrite this to Kotlin using ktor. This is what I've come up with so far: val serverCert = serverCertPath.inputStream().use { CertificateFactory.getInstance("X.509").generateCertificate(it) as X509Certificate } val keyStore = KeyStore.getInstance(...).apply { load(null, null) setCertificateEntry("serverCert", serverCert) } val trustManagerFactory = ... // init with keystore val sslContext = SSLContext.getInstance("TLS") // and init with above config val client = HttpClient(Java) { engine { config { sslContext(sslContext) } } } // So far so good. This server certificate config seems to work and cover the 'verify' parameter. Now for the other cert. val clientCert = CertificateFactory.getInstance("X.509").let { clientCertPath.inputStream().use { stream -> it.generateCertificate(stream) as X509Certificate } } client.request(url) { this.method = HttpMethod.Get // how to supply client cert? }  And now I'm stuck. How do I apply the client certificate to the request? Either client-level configuration or request-level configuration would be fine with me. Also, I still haven't used the pkeyPath. Where do I do that?

Answer: There are couple of ways to load pem files and creating a sslcontext out of it. However it is a bit verbose with traditional java. If you just want to go with that without the need for additional libraries, you can give the following stackoverflow topic a try How to build a SSLSocketFactory from PEM certificate and key without converting to keystore? Or else you can try my library which is a bit less verbose. What you need to do is load your pem files and create a key manager and trust manager which you can use to create a sslcontext. Ktor will be able to use this object. An example code snippet would be: var keyManager = PemUtils.loadIdentityMaterial("certificate-chain.pem", "private-key.pem"); var trustManager = PemUtils.loadTrustMaterial("some-trusted-certificate.pem"); var sslFactory = SSLFactory.builder() .withIdentityMaterial(keyManager) .withTrustMaterial(trustManager) .build(); var sslContext = sslFactory.getSslContext();  You can add the library with the following snippet to your project: implementation("io.github.hakky54:sslcontext-kickstart-for-pem:8.1.5")
Question: I often find I have something like this: cur = [0, 0] # the indices into array matrix = [[1,1,1]]  where I do matrix[cur[0]][cur[1]]  Is there any sort of unpacking syntax here? Like: matrix[*cur] 

Answer: If you switch to NumPy and you're using Python 3.11+, then yes, that works. import numpy as np cur = [0, 0] matrix = np.array([[1, 2, 3]]) print(matrix[*cur]) # -> 1  Before Python 3.11, you can just convert to tuple: print(matrix[tuple(cur)]) # -> 1  Based on the name matrix, NumPy might be a better solution in other ways too. For example you get elementwise operations.  Note: The Python 3.11 syntax change doesn't seem to be documented in the normal places (what's new and grammar). From a quick look, I only found it mentioned under typing.Unpack: "* couldn’t be used in certain places". It is covered in PEP 646 though, which introduced typing.TypeVarTuple.
Question: In the below code (Jupiter notebook), output is given wrong, how to correct the code? import numpy as np A = [1,22,231,1540] B = [[-1,1,0],[1,-1,1],[0,6,0],[0,6,0]] C = [[113401201],[10649],[1]] result = np.dot(np.dot(A, B),C) print(result)  output  [-1800609408]  The actual answer is  I want to find the error and correct it

Answer: You're probably running this code on 32-bit platform (where integers are 32 bit): A = np.array(A, dtype=np.int32) B = np.array(B, dtype=np.int32) C = np.array(C, dtype=np.int32) result = np.dot(np.dot(A, B), C) print(result)  Prints (incorrectly, because the value overflows): [-1800609408]   To correct it, use 64-bit values: A = np.array(A, dtype=np.int64) B = np.array(B, dtype=np.int64) C = np.array(C, dtype=np.int64) result = np.dot(np.dot(A, B), C) print(result)  Prints: [2494357888]
Question: I have a dataframe like below: data = [ [123456, "2017", 150.235], [123456, "2017", 160], [123456, "2017", 135], [123456, "2017", 135], [123456, "2017", 135], [123456, "2018", 202.5], [123456, "2019", 168.526], [123456, "2020", 175.559], [123456, "2020", 176], [123456, "2021", 206.667], [789101, "2017", 228.9], [789101, "2018", 208], [789101, "2018", 208], [789101, "2018", 208], ] df = pd.DataFrame( data, columns=[ "ID", "year", "value", ], ) df  In this dataframe I have an ID column and 2+ years. The year columns can contain 1 or more value columns. I would like to filter this dataframe so that all of the earliest year rows (even if there are duplicate values) and all of the latest year rows (again, even if there are duplicate values I want them). My desired output is:  I found another SO question that was similar: g = df.groupby("ID") (pd.concat([g.head(1), g.tail(1)]) .drop_duplicates() .sort_values('ID') .reset_index(drop=True))  but it only first to the first value within the first year and I want all of the values. Can anyone please advise?! Thank you !!

Answer: Try: out = df.groupby("ID", group_keys=False).apply( lambda x: x[(x.year == x.year.min()) | (x.year == x.year.max())] ) print(out)  Prints:  ID year value 0 123456 2017 150.235 1 123456 2017 160.000 2 123456 2017 135.000 3 123456 2017 135.000 4 123456 2017 135.000 9 123456 2021 206.667 10 789101 2017 228.900 11 789101 2018 208.000 12 789101 2018 208.000 13 789101 2018 208.000
Question: I have a problem whereby I want to filter the following dataframe such that I only return the rows where we have both a pie and a non-pie item:     ID set     1 apple pie, banana loaf   2 banana pie, apple pie   3 banana loaf, apple tart     Thus, the expected output would be:     ID set     1 apple pie, banana loaf     Note that every set in the set column contains exactly two items. What I have tried so far: df[(any("pie" in s for s in df['set'])) & (any("pie" not in s for s in df['set']))]  I expect I am doing something that is breaking Pandas dataframe filtering convention but not sure what exactly. Any help appreciated!

Answer: You could use apply on your dataframe: df[df.set.apply(lambda x: len([s for s in x if "pie" in s]) == 1)]  Results:  ID set 0 1 [apple pie, banana loaf]
Question: I'm transitioning from Spyder to VS Code, and I'm having an issue executing ad-hoc code on an interactive session. I first save this to a tmp1.py and run the selection on an interactive window: import pandas as pd df = pd.DataFrame({'a': [1,2,3,4], 'b': [1,1,2,3]})  Then on tmp2.py I just write and run: print(df)  It opens a new interactive window and complains NameError: name 'df' is not defined. How can I configure it to run on the currently active window / session? Note that I'm not using a Jupyter notebook but IPython. At some point I'll want to run code in parallel on several terminals, but at first I'd like to get this simple set-up working.

Answer: For some reason my "Jupyter > Interactive Window: Creation Mode" setting was "perfile", although "multiple" is supposed to be the default. After changing it to "multiple" everything works logically. I can also open sevaral interactive windows, and the selected code gets evaluated in the active one.
Question: How can I avoid calling the function I'm testing in two different places when writing table-driven tests where some of the tests should raise but others should not? This is what I want to do but it fails passing None to assertRaises: tests = [ (0, None), (1, None), (-1, TooFewException), (99, None), (100, TooManyException), ] for n, exc in tests: with self.assertRaises(exc): results = my_code(n) assert len(results) == n  The best I have come up with is this but the redundant call to my_code is bothering me: tests = [ (0, None), (1, None), (-1, TooFewException), (99, None), (100, TooManyException), ] for n, exc in tests: if exc is not None: with self.assertRaises(exc): my_code(n) else: results = my_code(n) assert len(results) == n   After adding a helper func on our base test case using the answer from @AmuroRay this is now: tests = [ (0, None), (1, None), (-1, TooFewException), (99, None), (100, TooManyException), ] for n, exc in tests: with self.assertRaisesUnlessNone(exc): results = my_code(n) assert len(results) == n 

Answer: contextlib has nullcontext() which can accomplish this. It's not really using fewer lines (without an ugly one-liner branch), but it eliminates the second user-code call: from contextlib import nullcontext for n, exc in tests: if exc is None: cm = nullcontext() else: cm = self.assertRaises(exc) with cm: results = my_code(n) assert len(results) == n
Question: is there any way we can format a big number in tqdm's progress bar for better readability? For example, 1000000row/s must be 1,000,000rows/s . I know about unit scale, but it formats a number like 1,21M rows/s which is not currently what I am trying to achieve Thanks!

Answer: A little bit of reverse-engineering how tqdm handles it: from time import sleep from tqdm import tqdm tqdm.format_sizeof = lambda x, divisor=None: f"{x:,}" if divisor else f"{x:5.2f}" with tqdm(total=1_000_000, unit_scale=True) as t: for _ in range(1_000_000): sleep(0.00001) t.update()  Prints:  8%|████████████▊ | 78,888/1,000,000 [00:04<00:56, 0.00it/s]
Question: I am trying to use panads to manage data that I am pulling from an API. The data has one column that is a nested list and I am trying to extract it out to their own columns. The data looks like this so far:     id mail displayName propertiesRegistered createdDateTime     00000000-0000-0000-0000-000000000000 joe.user@email.com User, Joe ['address', 'mobilePhone', 'officePhone'] 2023-08-19T15:00:00.00Z     The desired output would look like:     id mail displayName address mobilePhone officePhone homePhone createdDateTime     00000000-0000-0000-0000-000000000000 joe.user@email.com User, Joe TRUE TRUE TRUE FALSE 2023-08-19T15:00:00.00Z     I've tried expanding, series, and pivot tables but cannot seem to figure it out. And not sure I'm even phrasing my question correctly in my searching. A lot of people and examples have the data made into additional rows, which I was able to do, but getting it to a single row is ideal. Any help is greatly appreciated.

Answer: You can use str.get_dummies: valid_properties = ['address', 'mobilePhone', 'officePhone', 'homePhone'] df = df.join(df.pop('propertiesRegistered').agg('|'.join).str.get_dummies() .reindex(columns=valid_properties, fill_value=0) # or astype(bool) for real booleans .replace({1: 'TRUE', 0: 'FALSE'}) )  Or a crosstab: s = df.pop('propertiesRegistered').explode() df = df.join(pd.crosstab(s.index, s) .reindex(columns=valid_properties, fill_value=0) .gt(0) )  Output:  id mail displayName createdDateTime address mobilePhone officePhone homePhone 0 00000000-0000-0000-0000-000000000000 joe.user@email.com User, Joe 2023-08-19T15:00:00.00Z TRUE TRUE TRUE FALSE
Question: I need to convert data frame column to int. df['Slot'].unique()  displays this: array(['1', '2', '3', '4', 1, 3, 5], dtype=object)  some values have '' around it some dont. I tried to convert the data type to int as below: df['Slot']=df['Slot'].astype('Int64')  I get this error: TypeError: cannot safely cast non-equivalent object to int64  Any ideas how to convert to int?

Answer: You should first convert to numeric, then cast to Int64: df['Slot'] = pd.to_numeric(df['Slot']).astype('Int64')  After casting the type: df.dtypes Slot Int64 dtype: object  As noted in comments, if you use numpy's int64 type this works from scratch: df['Slot'].astype('int64')