Repository: WeblateOrg/weblate

Issue:
Statistics fail to update
### Describe the issue

The background job updating the stats is failing with the following error:

Failure while executing task: KeyError: 'stats_timestamp'

### I already tried

- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar issues in this repository.

### Steps to reproduce the behavior

Update a translation.

### Expected behavior

_No response_

### Screenshots

_No response_

### Exception traceback

```pytb
weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,727: ERROR/1511] Failure while executing task: KeyError: 'stats_timestamp'                                                              weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,727: ERROR/ForkPoolWorker-8] Failure while executing task: KeyError: 'stats_timestamp'                                                  
weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/1511] Failure while executing task: KeyError: {'signal': <Signal: task_failure providing_args={'exception', 'task_id', 'traceback', 'args', 'kwargs', 'einfo'}>, 'sender': <@task: weblate.utils.tasks.update_translation_stats_parents of weblate at 0x7fc9e3c49210>, 'task_id': '41c6e00c-36ec-4f25-99c4-35b7e471fa56', 'args': [7621], 'kwargs'
: {}, 'traceback': <traceback object at 0x7fc9cd31e240>, 'einfo': <ExceptionInfo: ExceptionWithTraceback()>}                                                                                                         weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/ForkPoolWorker-8] Failure while executing task: KeyError: {'signal': <Signal: task_failure providing_args={'exception', 'task
_id', 'traceback', 'args', 'kwargs', 'einfo'}>, 'sender': <@task: weblate.utils.tasks.update_translation_stats_parents of weblate at 0x7fc9e3c49210>, 'task_id': '41c6e00c-36ec-4f25-99c4-35b7e471fa56', 'args': [7621], 'kwargs': {}, 'traceback': <traceback object at 0x7fc9cd31e240>, 'einfo': <ExceptionInfo: ExceptionWithTraceback()>}                                                                                             weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/1511] Failure while executing task                                                                                           weblate-docker-weblate-1       | celery-celery stderr | Traceback (most recent call last):                                                                                                                           weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task                                                                weblate-docker-weblate-1       | celery-celery stderr |     R = retval = fun(*args, **kwargs)                                                                                                                        weblate-docker-weblate-1       | celery-celery stderr |                  ^^^^^^^^^^^^^^^^^^^^                                                                                                                        weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__                                                        weblate-docker-weblate-1       | celery-celery stderr |     return self.run(*args, **kwargs)                                                                                                                         
weblate-docker-weblate-1       | celery-celery stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                         weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/tasks.py", line 79, in update_translation_stats_parents                                        
weblate-docker-weblate-1       | celery-celery stderr |     translation.stats.update_parents()                                                                                                                       weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 312, in update_parents                                                         
weblate-docker-weblate-1       | celery-celery stderr |     stat.update_stats()                                                                                                                                      weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 329, in update_stats                                                           
weblate-docker-weblate-1       | celery-celery stderr |     self.calculate_basic()                                                                                                                                   weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 337, in calculate_basic                                                        
weblate-docker-weblate-1       | celery-celery stderr |     self._calculate_basic()                                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 827, in _calculate_basic                                                       
weblate-docker-weblate-1       | celery-celery stderr |     stats[item] = max(values, default=time.time())                                                                                                           weblate-docker-weblate-1       | celery-celery stderr |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                           
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 824, in <genexpr>                                                              weblate-docker-weblate-1       | celery-celery stderr |     values = (stats_obj.aggregate_get(item) for stats_obj in all_stats)                                                                                      
weblate-docker-weblate-1       | celery-celery stderr |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 225, in aggregate_get                                                          weblate-docker-weblate-1       | celery-celery stderr |     return self._data[name]                                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr |            ~~~~~~~~~~^^^^^^                                                                                                                                  
weblate-docker-weblate-1       | celery-celery stderr | KeyError: 'stats_timestamp'                                                                                                                                  weblate-docker-weblate-1       | celery-celery stderr | [2023-11-29 12:29:32,728: ERROR/ForkPoolWorker-8] Failure while executing task                                                                               
weblate-docker-weblate-1       | celery-celery stderr | Traceback (most recent call last):                                                                                                                           weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task                                                                
weblate-docker-weblate-1       | celery-celery stderr |     R = retval = fun(*args, **kwargs)                                                                                                                        weblate-docker-weblate-1       | celery-celery stderr |                  ^^^^^^^^^^^^^^^^^^^^                                                                                                                        
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__                                                        weblate-docker-weblate-1       | celery-celery stderr |     return self.run(*args, **kwargs)                                                                                                                         
weblate-docker-weblate-1       | celery-celery stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                         weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/tasks.py", line 79, in update_translation_stats_parents                                        
weblate-docker-weblate-1       | celery-celery stderr |     translation.stats.update_parents()                                                                                                                       weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 312, in update_parents                                                         
weblate-docker-weblate-1       | celery-celery stderr |     stat.update_stats()
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 329, in update_stats
weblate-docker-weblate-1       | celery-celery stderr |     self.calculate_basic()
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 337, in calculate_basic
weblate-docker-weblate-1       | celery-celery stderr |     self._calculate_basic()
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 827, in _calculate_basic
weblate-docker-weblate-1       | celery-celery stderr |     stats[item] = max(values, default=time.time()) 
weblate-docker-weblate-1       | celery-celery stderr |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 824, in <genexpr>
weblate-docker-weblate-1       | celery-celery stderr |     values = (stats_obj.aggregate_get(item) for stats_obj in all_stats)
weblate-docker-weblate-1       | celery-celery stderr |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weblate-docker-weblate-1       | celery-celery stderr |   File "/usr/local/lib/python3.11/site-packages/weblate/utils/stats.py", line 225, in aggregate_get
weblate-docker-weblate-1       | celery-celery stderr |     return self._data[name]
weblate-docker-weblate-1       | celery-celery stderr |            ~~~~~~~~~~^^^^^^
weblate-docker-weblate-1       | celery-celery stderr | KeyError: 'stats_timestamp'
```


### How do you run Weblate?

Docker container

### Weblate versions

weblate:bleeding (48863d1f4b3)

### Weblate deploy checks

_No response_

### Additional context

Related to https://github.com/WeblateOrg/weblate/issues/10473


Discussion:
Thanks a lot!
Thank you for your report; the issue you have reported has just been fixed.

* In case you see a problem with the fix, please comment on this issue.
* In case you see a similar problem, please open a separate issue.
* If you are happy with the outcome, don’t hesitate to support Weblate by making a [donation](https://weblate.org/donate/).
Even with the fix from #10508 (using bleeding docker image with HEAD 2f6cb95738f68b92c8e) we are still seeing:
![image](https://github.com/WeblateOrg/weblate/assets/671757/6669944e-8424-4263-a16c-5876faa934db)
![image](https://github.com/WeblateOrg/weblate/assets/671757/e3af6420-607d-45a9-9521-0fbd9750ab90)

And clicking the "1707 unfinished" link leads us to "No strings found!"
The stats will update only upon translation change, not automatically.
Yes, I have updated a translation, also flag one as "need editing".  But still marked as 90% translated.
None of the "Unfinished" number is correct.
https://translate.koha-community.org/projects/koha/master/#languages

French: 1731, but if you click on this number you get 2090
Spanish: 1707, 0
German: 1709, 217
Greek: 9673, 14752
and so on...
Do you still see some errors in the log? I've just pushed 3537b3dfdc so that stats age is visible on the page, that would give us a clue whether it is wrongly calculated or update was skipped.
No error found in the log.

I will wait for the next docker image to be build and I will update!
Another weirdness:
https://translate.koha-community.org/projects/koha/23-05/#languages (another category: "23-05" instead of "master").

German is 100% translated, it has the green tick. On hover text is: "1859 strings".
On the component list view: https://translate.koha-community.org/projects/koha/23-05/-/de/
1859 is the number for the first component "installer".

This "installer" component has the "source code repo" linked to the remote git repo, whereas other components within this 23-05 category point to it using weblate://koha/23-05/installer
Debug commit applied (docker bleeding 423bbe55b4)

Numbers are still wrong.

Master/Spanish components have been updated "6 hours ago"
https://translate.koha-community.org/projects/koha/master/installer/es/#information
Same for 23-05/German
https://translate.koha-community.org/projects/koha/23-05/installer/de/#information

Note that "0" is displayed when at the category level (but I guess it's expected):
https://translate.koha-community.org/projects/koha/23-05/-/de/#information
0 is not expected, that means that the stats were not updated with the new code yet. So there is clearly an issue with updating category/language stats.

Resolving patch:
diff --git a/weblate/utils/stats.py b/weblate/utils/stats.py
--- a/weblate/utils/stats.py
+++ b/weblate/utils/stats.py
@@ -443,23 +443,27 @@ def save(self, update_parents: bool = True):
                     lambda: update_translation_stats_parents.delay(pk)
                 )
 
-    def get_project_language_stats(self):
-        project_language = ProjectLanguage(
-            project=self._object.component.project, language=self._object.language
-        )
-        return project_language.stats
+    def get_update_objects(self, *, full: bool = True):
+        translation = self._object
+        component = translation.component
 
-    def get_update_objects(self):
         # Language
-        yield self._object.language.stats
+        yield translation.language.stats
 
         # Project / language
-        yield self.get_project_language_stats()
+        yield component.project.stats.get_single_language_stats(translation.language)
+
+        # Category / language
+        category = component.category
+        while category:
+            yield category.stats.get_single_language_stats(translation.language)
+            category = category.category
 
-        # Component
-        yield self._object.component.stats
-        # Component list, category, project and global
-        yield from self._object.component.stats.get_update_objects()
+        if full:
+            # Component
+            yield component.stats
+            # Component list, category, project and global
+            yield from component.stats.get_update_objects()
 
     @property
     def language(self):
@@ -911,8 +915,7 @@ def update_language_stats(self):
         # Update languages
         for translation in prefetch_stats(self.get_child_objects()):
             translation.stats.update_stats(update_parents=False)
-            extras.append(translation.language.stats)
-            extras.append(translation.stats.get_project_language_stats())
+            extras.extend(translation.stats.get_update_objects(full=False))
 
         # Update our stats
         self.update_stats()


Test patch: