Repository: scrapy/scrapy

Issue:
Add logging functionality to `memusage` extension


## Summary

To add logging functionality to memusage extension.

## Motivation

Scrapy jobs with `MEMUSAGE_ENABLED : True` and defined `MEMUSAGE_LIMIT_MB`  (all jobs on scrapy cloud) can be stopped early due to overuse of RAM memory and receive  `memusage_exceeded` outcome.

First thing required to debug RAM memory leaks - is to identify.. pattern of RAM memory usage.
Is RAM usage continuously increased at higher rates during runtime?
or Is RAM usage rapidly increased over limit in last several minutes after hours or even days of stable runtime performance?
Each reason require different approaches to debug RAM memory leaks.

It will be much easier to debug this if  value of `self.get_virtual_size()` will be added to log in  `_check_limit` method of `memusage` extension
https://github.com/scrapy/scrapy/blob/6ded3cf4cd134b615239babe28bb28c3ff524b05/scrapy/extensions/memusage.py#L77-L89

## Describe alternatives you've considered

Applying `MEMUSAGE_WARNING_MB` setting to ~80-90% of `MEMUSAGE_LIMIT_MB` - current implementation of `memusage` extension warns only 1 time so it is not enough data for this.

Manually subclass `memusage` extension with similar changes - as well as any other option it will require to reschedule job. It may be not siutable for jobs with several days(and more) total runtime. So from this side it is preferable that it will be applied in scrapy itself and with enabled this loggin by default.
 
## Additional context

Similar functionality previously requested here https://github.com/scrapy/scrapy/issues/2173



Discussion:
I have a patch ready to address this but don't have permission to push my branch. Am I doing something wrong? 
@JazzGlobal I suppose you're trying to push to the main Scrapy repository, you should instead [fork the repo](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks) and [create a pull request from that fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).
> @JazzGlobal I suppose you're trying to push to the main Scrapy repository, you should instead [fork the repo](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks) and [create a pull request from that fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).

Ah, I sure was. Thank you for the prompt and helpful response!

Resolving patch:
diff --git a/scrapy/extensions/memusage.py b/scrapy/extensions/memusage.py
--- a/scrapy/extensions/memusage.py
+++ b/scrapy/extensions/memusage.py
@@ -75,7 +75,8 @@ def update(self):
         self.crawler.stats.max_value('memusage/max', self.get_virtual_size())
 
     def _check_limit(self):
-        if self.get_virtual_size() > self.limit:
+        current_mem_usage = self.get_virtual_size()
+        if current_mem_usage > self.limit:
             self.crawler.stats.set_value('memusage/limit_reached', 1)
             mem = self.limit / 1024 / 1024
             logger.error("Memory usage exceeded %(memusage)dM. Shutting down Scrapy...",
@@ -92,6 +93,8 @@ def _check_limit(self):
                 self.crawler.engine.close_spider(self.crawler.engine.spider, 'memusage_exceeded')
             else:
                 self.crawler.stop()
+        else:
+            logger.info("Current memory usage is %(virtualsize)dM", {'virtualsize': current_mem_usage / 1024 / 1024})
 
     def _check_warning(self):
         if self.warned:  # warn only once


Test patch: