eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dataframe and I wanted to apply a certain function on a set of columns. Something like: data[["A","B","C","D","E"]].apply(some_func, axis=1)  In the some_func function, the first step is extracting out all the column values into separate variables. def some_func(x): a,b,c,d,e = x # or x.tolist() #Some more processing  To reproduce, the result, use x = pd.Series([1,2,3,4,5], index=["A","B","C","D","E"])  Now, my question is, why does %%timeit a,b,c,d,e = x.tolist()  Output: 538 ns ± 2.82 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  perform better than %%timeit a,b,c,d,e = x  Output: 1.61 µs ± 15.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) 

Answer: Let's define two functions and inspect them with dis: from dis import dis from pandas import Series x = Series([1,2,3,4,5], index=["A","B","C","D","E"]) def a(): a, b, c, d, e = x.tolist() def b(): a, b, c, d, e = x dis(a) dis(b)  Executing the above will yield: # dis(a) 7 0 LOAD_GLOBAL 0 (x) 2 LOAD_METHOD 1 (tolist) 4 CALL_METHOD 0 6 UNPACK_SEQUENCE 5 8 STORE_FAST 0 (a) 10 STORE_FAST 1 (b) 12 STORE_FAST 2 (c) 14 STORE_FAST 3 (d) 16 STORE_FAST 4 (e) 18 LOAD_CONST 0 (None) 20 RETURN_VALUE # dis(b) 10 0 LOAD_GLOBAL 0 (x) 2 UNPACK_SEQUENCE 5 4 STORE_FAST 0 (a) 6 STORE_FAST 1 (b) 8 STORE_FAST 2 (c) 10 STORE_FAST 3 (d) 12 STORE_FAST 4 (e) 14 LOAD_CONST 0 (None) 16 RETURN_VALUE  From the above, it seems that, if anything, function (a) has more instructions. So why is it faster? As explained in this answer, looking at the contents of UNPACK_SEQUENCE, one can see that there are some special-cases, such as when the number of left-hand side variables is equal to the length of the right-hand side object. So, x.tolist() under the hood uses numpy method to create a list from the array data, which allows making use of the optimization for this special case (you can check the deterioration in performance by changing the number of arguments on the left-hand side, e.g. a, *b = range(3), will work, but will be slower than a, b, c = range(3)). When the right-hand side object is not a Python tuple or a list, then Python iterates over the contents of the object, which appears to be less efficient. For practical reasons, if you really want best performance (with the current versions of the modules), you can swap x.tolist() with x._values.tolist(), which should give about 10-15% boost in performance (you're just removing one layer of pandas to numpy call, and doing it directly here). The caveat is that these types of optimizations are sensitive to what's happening in lower-level code, so there is no guarantee that performance gains will be there in future Python/library combinations.
Question: I have a python file with some code and simple tests on that code. I would like to invoke pytest from that python file, have it collect only the tests in that file, and run them. For example, foo.py: # ... various code above... def test_foo(): foo = Foo() assert foo() def test_bar(): bar = Bar() assert bar.baz if __name__ == '__main__': import pytest pytest.main()  I now would like to run python foo.py and have pytest run the two tests in that file, and those tests only. However, when I run python foo.py, pytest collects ALL tests from all modules that are rooted from the path in which I ran the command. How can I run python foo.py and have the pytest.main() invoked in foo.py only call the tests in foo.py and no others?

Answer: According to the pytest documentation, options and arguments can be passed to pytest.main. To run tests in foo.py, this would work: # ... various code above... def test_foo(): foo = Foo() assert foo() def test_bar(): bar = Bar() assert bar.baz if __name__ == '__main__': import pytest pytest.main(["foo.py"]) # consider using below # pytest.main([__file__])
Question: I want to create on Abstract Model class for future inheriting like this: class AbstractModel(models.Model): created_at = models.DateTimeField( auto_now_add=True, blank=True, null=True, ) created_by = models.ForeignKey( settings.AUTH_USER_MODEL, on_delete=models.SET_NULL, related_name='XXX_created_by', blank=True, null=True, ) class Meta: abstract = True  Field 'created_at' is working fine, but how to generate related_name in 'created_by' for my child classes to prevent clashing?

Answer: As the Be careful with related_name and related_query_name section of the documentation says, you can:  To work around this problem, when you are using related_name or related_query_name in an abstract base class (only), part of the value should contain '%(app_label)s' and '%(class)s'.  '%(class)s' is replaced by the lowercased name of the child class that the field is used in.  '%(app_label)s' is replaced by the lowercased name of the app the child class is contained within. Each installed application name must be unique and the model class names within each app must also be unique, therefore the resulting name will end up being different.    You thus can work with: class AbstractModel(models.Model): # … created_by = models.ForeignKey( settings.AUTH_USER_MODEL, on_delete=models.SET_NULL, related_name='%(class)s_created_by', blank=True, null=True, ) class Meta: abstract = True Then the related_name will be foo_created_by if the name of the model that inherits is named foo. Or if the same model name can occur in different apps: class AbstractModel(models.Model): # … created_by = models.ForeignKey( settings.AUTH_USER_MODEL, on_delete=models.SET_NULL, related_name='%(app_label)s_%(class)s_created_by', blank=True, null=True, ) class Meta: abstract = True Then the related_name will be bar_foo_created_by if the name of the model that inherits is named foo in an app named bar.
Question: I have a Django Subquery which returns a Sum. If the subquery finds at least one result, it works fine. But, if the subquery finds no records, it returns None which then causes any other calculations using this result (I use it later in my query in an F expression) to also result in None. I am trying to Sum all non-null 'consumed' values - sometimes, they are all null and therefore there are no rows upon which to sum. I would like this to result in a 0 instead of None ... annotate(tapx=Subquery(InvTrx.objects.filter(job=OuterRef('job')).\ filter(consumed__isnull=False).\ filter(inventory__inv_type='APX').\ values('job__job').\ annotate(tot_cons=Sum('consumed', default=0)).\ values('tot_cons') )).\ ...  I've tried Coalesce with and without Value(0) annotate(tot_cons=Coalesce(Sum('consumed', default=0)), 0).\ annotate(tot_cons=Coalesce(Sum('consumed', default=0)), Value(0)).\  the value of tapx (which I reuse in F expressions in another part of the query) = None if no rows are returned. If at least one row is returned, this works fine. If no rows are returned, I would like the value of tapx to be 0 instead of None so that the value of fg_total in the following annotation results in a number and not None: annotate(fg_total=F('fg') + F('tapx'))  Doing this outside a subquery, I have used "or 0" to force the value to 0 if the result is None - is there a way to do this inside a subquery? I also tried, when I do use the result of the Subquery in an F expression, to use a conditional like so: annotate(fg_total=F('fg') + Case(When(~Exists('tapx'), then=F('tapx')),default=Value(0)))  but this results in an error - 'str' object has no attribute 'order_by' (I am not sure what this means in this context since I am not using 'order_by' anywhere?) I am using Django 3.0. I'm stuck.

Answer: You should not Coalesce the .annotate(…) in the Subquery, but the result of the Subquery, so: .annotate(tapx=Coalesce( Subquery(InvTrx.objects.filter( job=OuterRef('job'), consumed__isnull=False, inventory__inv_type='APX' ).values('job_id').annotate(tot_cons=Sum('consumed')).values('tot_cons')), Value(0)) )
Question: I'm stuck on thinking about implementing a "only one entry might be True for one combination". A Project has n members (Guards) through an intermediate table.  every Guard may be member of n Projects only one combination of Guard <-> Project is allowed (unique_together) a MemberShip might be the 'Main' one (is_main) BUT: Only one of the memberships may be Main.  Do I oversee something or do I have to implement a custom validation on my own? To complete this, see the given Model: class Project(models.Model): client = models.ForeignKey(Client, on_delete=models.CASCADE) shortname = models.CharField(_('shortname'), max_length=50) description = models.TextField(_('description'), blank=True) members = models.ManyToManyField(Guard, through='ProjectMembership') class Meta: unique_together = ['client', 'shortname'] class ProjectMembership(models.Model): guard = models.ForeignKey(Guard, on_delete=models.CASCADE) project = models.ForeignKey(Project, on_delete=models.CASCADE) is_main = models.BooleanField(_('is main project'), default=False) class Meta: unique_together = ['guard', 'project'] 

Answer: You can work with a UniqueConstraint [Django-doc] that is filtered: from django.db.models import UniqueConstraint, Q class ProjectMembership(models.Model): guard = models.ForeignKey(Guard, on_delete=models.CASCADE) project = models.ForeignKey(Project, on_delete=models.CASCADE) is_main = models.BooleanField(_('is main project'), default=False) class Meta: constraints = [ UniqueConstraint(fields=('guard', 'project'), name='unique_guard'), UniqueConstraint(fields=('guard',), condition=Q(is_main=True), name='one_main_project_per_guard'), ] Here we thus ensure that if we filter ProjectMembership for is_main=True, that the set of guards is unique, hence a certain Guard can only occur once for is_main, and this thus means that a Guard has at most one Project for which is_main is True.   Note: As the documentation on unique_together [Django-doc] says, the unique_together constraint will likely become deprecated. The documentation advises to use the UniqueConstraint [Django-doc] from Django's constraint framework.
Question: If all the fastapi endpoints are defined as async def, then there will only be 1 thread that is running right? (assuming a single uvicorn worker). Just wanted to confirm in such a setup, we will never hit the python's Global Interpreter Lock. If the same was to be done in a flask framework with multiple threads for the single gunicorn worker, then we would be facing the GIL which hinders the true parallelism between threads. So basically, in the above fastapi, the parallelism is limited to 1 since there is only one thread. And to make use of all the cores, we would need to increase the number of workers either using gunicorn or uvicorn. Is my understanding correct?

Answer: Your understanding is correct. When using 1 worker with uvicorn, only one process is run. That means, there is only one thread that can take a lock on the interpreter that is running your application. Due to the asynchronous nature of your FastAPI app, it will be able to handle multiple simultaneous requests, but not in parallel. If you want multiple instances of your application run in parallel, you can increase your workers. This will spin up multiple processes (all single threaded as above) and Uvicorn will distribute the requests among them. Note that you cannot have shared global variables across workers. These are separate instances of your FastAPI app and do not communicate with each other. See this answer for more info on that and how to use databases or caches to work around that.
Question: The following code print 2022-01-01 05:30:00-04:56. What's -04:56? import pytz, datetime tz = pytz.timezone("America/New_York") t = datetime.datetime(2022, 1,1, 5, 30) u = t.replace(tzinfo=tz) print(u) 2022-01-01 05:30:00-04:56  In jupyter, u has the value of datetime.datetime(2022, 1, 1, 5, 30, tzinfo=<DstTzInfo 'America/New_York' LMT-1 day, 19:04:00 STD>). What's DstTzInfo? And what's 19:04:00 STD?

Answer: (Delete previous answer) My bad: I parsed datetime.datetime(2022, 1,1, 5, 30) as 30 May (I'm answering this on 10 June.). We're not in daylight savings time in January. ^ For me, this whole question is yet another reminder that I don't * really * understand time. OP. will you please remove this as an accepted solution? (It will not allow me to delete an accepted solution.)
Question: I'm writing a site on Flask and I came across the problem that when sending emails to email users there is such an error. smtplib.SMTPAuthenticationError: (535, b'5.7.8 Username and Password not accepted. Learn more at\n5.7.8 https://support.google.com/mail/?p=BadCredentials i1-20020ac25221000000b00478f5d3de95sm4732790lfl.120 - gsmtp')  I searched Google for a solution to the problem, but it said that you need to disable a feature that has not been supported by GMAIL recently. Maybe someone knows how to solve this problem now? Here is my config connection: app.config['MAIL_SERVER'] = 'smtp.googlemail.com' app.config['MAIL_PORT'] = 587 app.config['MAIL_USE_TLS'] = True app.config['MAIL_USERNAME'] = os.getenv('MAIL_USERNAME') app.config['MAIL_PASSWORD'] = os.getenv('MAIL_PASSWORD')  Help me please

Answer: Since You can't use the less secure app now as the deadline was 30th May 2022. An alternative way to send Gmail via flask using an App password Before generating the APP password your google account must enable the 2FA. If the 2FA is enabled then you can hop on to the security section in the manage accounts and there you can see the APP password section. For generating the APP password you also can read -> here The below code I tried and it's working I have used python another dependencies pip install Flask-Mail  from flask import Flask from flask_mail import Mail, Message app = Flask(__name__) mail= Mail(app) app.config['MAIL_SERVER']='smtp.gmail.com' app.config['MAIL_PORT'] = 465 app.config['MAIL_USERNAME'] = 'someone@gmail.com' app.config['MAIL_PASSWORD'] = 'YOUR_APP_PASSWORD' app.config['MAIL_USE_TLS'] = False app.config['MAIL_USE_SSL'] = True mail = Mail(app) @app.route("/") def index(): msg = Message('Hello', sender = 'someone@gmail.com', recipients = ['someone@gmail.com']) msg.body = "Hello Flask message sent from Flask-Mail" mail.send(msg) return "Sent" if __name__ == '__main__': app.run(debug = True)
Question: I recently was using smtp library for sending emails from gmail account but recently it stopped working after research I found out the google can not let you enable the less secure app anymore . So is there any workaround this ? 

Answer: If you want to continue using Gmail SMTP, you can set it up by setting an app password. An app password works like an alternate password for your account. It can only be used by the applications you share it with, so it’s more secure than sharing your primary password. Here's how you can set it up: https://support.google.com/accounts/answer/185833?hl=en
Question: I'm not sure if node.js 18 supports centos 7 and is it a requirement to install python 3 for node.js 18?

Answer: Step 1 - curl --silent --location https://rpm.nodesource.com/setup_18.x | sudo bash - Step 2 - sudo yum -y install nodejs I don't think you need Python 3. Reference - https://computingforgeeks.com/install-node-js-on-centos-rhel-rocky-linux/
Question: I now have the updated code as follows:  # Hyperparameters random_seed = 123 learning_rate = 0.01 num_epochs = 10 batch_size = 128  device = torch.device("cuda:1" if torch.cuda.is_available() else "cpu") for epoch in range(num_epochs): model = resnet34.train() for batch_idx, (features, targets) in enumerate(train_generator):  features = features.to(device) targets = targets.to(device) ### FORWARD AND BACK PROP logits = model(features) cost = torch.nn.functional.cross_entropy(logits, targets) optimizer.zero_grad() cost.backward() ### UPDATE MODEL PARAMETERS optimizer.step() ### LOGGING if not batch_idx % 50: print ('Epoch: %03d/%03d | Batch %03d/%03d | Cost: %.4f' %(epoch+1, num_epochs, batch_idx, len(datagen)//batch_size, cost)) model = model.eval() # eval mode to prevent upd. batchnorm params during inference with torch.set_grad_enabled(False): # save memory during inference print('Epoch: %03d/%03d training accuracy: %.2f%%' % ( epoch+1, num_epochs, compute_accuracy(model, train_generator)))  When having only one image, the code runs fine. But, when I add another image or more, I get the following: features = features.to(device) targets = targets.to(device) AttributeError: 'numpy.ndarray' object has no attribute 'to' 

Answer: It would be nice to see your train_generator code for clarity, but it does not seem to be a torch DataLoader. In this case, you should probably convert your arrays to tensors manually. There are several ways to do so:  torch.from_numpy(numpy_array) - for numpy arrays; torch.as_tensor(list) - for common lists and tuples; torch.tensor(array) should also work but the above ways will avoid copying the data when possible.
Question: What is the best way to convert a Pandas series that contains strings of the type "10%" and "0.10" into numeric values? I know that if I have a series with just "0.10" type strings I can just do pd.to_numeric. I also know that if I have a series of "10%" type strings I can do str.replace("%","") and then do pd.to_numeric and divide by 100. The issue I have is for a series with a mix of "0.10" and "10%" type strings. How do I best convert this into a series with the correct numeric types. I think I could do it by first making a temporary series with True / False depending on if the string has "%" in it or not and then based on that applying a function. But this seems inefficient. Is there a better way? What I Have Tried for Reference: mixed = pd.Series(["10%","0.10","5.5%","0.02563"]) mixed.str.replace("%","").astype("float")/100 0 0.100000 1 0.001000 2 0.055000 3 0.000256 dtype: float64 # This doesn't work, because even the 0.10 and 0.02563 are divided by 100. 

Answer: The easiest solution is to select entries using a mask and handle them in bulk: from pandas import Series, to_numeric mixed = Series(["10%", "0.10", "5.5%", "0.02563"]) # make an empty series with similar shape and dtype float converted = Series(index=mixed.index, dtype='float') # use a mask to select specific entries mask = mixed.str.contains("%") converted.loc[mask] = to_numeric(mixed.loc[mask].str.replace("%", "")) / 100 converted.loc[~mask] = to_numeric(mixed.loc[~mask]) print(converted) # 0 0.10000 # 1 0.10000 # 2 0.05500 # 3 0.02563 # dtype: float64
Question: If I have a python generator function, let's say this one: def gen(): x = 0 while (true): yield x x += 1  This function remembers its current state, and every time you call gen(), yields a new value. Essentially, I would like a Kotlin sequence which can remember its state.

Answer:  def gen(): x = 0 while (true): yield x x += 1  This function remembers its current state, and every time you call gen(), yields a new value.  This is incorrect. Every time you call gen() you get a new "generator object" whose state is independent of any other generator object created by this function. You then query the generator object to get the next number. For example: def demo(): numbers = gen() # 'gen()' from your question for _ in range(0, 3): next_number = next(numbers) print(next_number) if __name__ == '__main__' demo() print() demo()  Output: 0 1 2 0 1 2  As you can see, the sequence of numbers "starts over" when you call gen() again (though if you kept a reference to the old generator object it would continue from 2, even after calling gen() again). In Kotlin, you can use the kotlin.sequences.iterator function. It creates an Iterator which lazily yields the next value, just like a Python generator object. For example: fun gen() = iterator { var x = 0 while (true) { yield(x) x++ } } fun demo() { val numbers = gen() repeat(3) { val nextNumber = numbers.next() println(nextNumber) } } fun main() { demo() println() demo() }  Which will output: 0 1 2 0 1 2  Just like the Python code. Note you can do the essentially the same thing with a Kotlin Sequence, you just have to convert the Sequence into an Iterator if you want to use it like a Python generator object. Though keep in mind that Kotlin sequences are meant more for defining a series of operations and then lazily processing a group of elements in one go (sort of like Java streams, if you're familiar with them).
Question: I have this type: class SomeResource: id: int name: str  And I need this type: class SomeResourceQuery: id: Optional[int] name: Optional[str]  But I'd like to avoid having to write it by hand. Is it possible to generate this SomeResourceQuery type from the SomeResource type? Just convert all the types of the fields to optional. (Update: Already optional fields can stay optional - no nested optionals.) I plan to use this SomeResourceQuery in a repository, like this: class SomeResourceRepository: def get_one_or_none(self, query: SomeResourceQuery) -> Optional[SomeResource]: ...   Update: Just to show what I'm thinking currently: class SomeResource: id: int name: str # I don't want to write this by hand: # class SomeResourceQuery: # id: Optional[int] # name: Optional[str] # So what can I do here to make all fields that are not already optional, optional? SomeResourceQuery = type("SomeResourceQuery", SomeResource) # What to do here? 

Answer: You can use the type constructor to construct the new type with the appropriate annotations. def construct_query_class(cls: type) -> type: annotations = {key: typing.Optional[value] for key, value in cls.__annotations__.items()} return dataclasses.dataclass(type(cls.__name__ + 'Query', (), {'__annotations__': annotations})) class SomeResource: id: int name: str SomeResourceQuery = construct_query_class(SomeResource) # type: typing.Any
Question:     Index E_Id P_Id Date     121 701 9002 2021   122 701 9001 2019   123 702 9002 2021   124 702 9002 2019   125 703 9001 2021   126 704 9002 2019   127 704 9003 2019     Now I want to Create another DataFrame groupedby E_Id But I want to Count the number of rows against each P_Id call it 'x', and then sum of 'x' for whoever is linked with each E_Id So     E_Id TotalOfPIds     701 6     Can anyone help me? As an intermediary step, I did this: data['_Pid_Total'] = data.groupby('P_Id')[['P_Id']].transform('count') And then for single E_Id it works like this: data.loc[data['E_Id'] == '701', ['E_Id', 'P_Id', '_Pid_Total']].groupby(['E_Id', 'P_Id']).first().sum() returning a single integer. However I want to use this in Transform method or just do it for entire DataFrame.

Answer: Thank you for an interesting question. My approach is:  You creates a df1 with unique combination of E_id and P_id Create df2 to count number of unique occur of P_id Merge df1 and df2, then use groupby with sum  I modify your sample a bit # Dataframe: df = pd.DataFrame({'e':[701, 701,701,701, 701, 701,702,702,703,704, 704, 704], 'p':[9002, 9002, 9002, 9002, 9001, 9003, 9002, 9002, 9001, 9002, 9003, 9001]}) # Step 1: Unique combination df1 = df.groupby(['e', 'p']).count().reset_index() # Step 2: Count Number of occurs for each unique value P_id df2 = df.groupby('p').count().reset_index().rename(columns={'e':'val'}) # Step 3: Merge then use groupby and sum df3 = pd.merge(df1, df2, on=['p'], how='left') df3.groupby('e')[['val']].sum().reset_index()