Repository: python-babel/babel

Issue:
extract: Wrong lineno when arguments are spread over multiple lines and contain a function call
## Overview

Consider this piece of code:
```py
_(
    "hello",
    dummy_fn_call()
)
```
Extracting it yields the following result:
```py
(3, 'hello', [], None)
```
As you may have noticed, the lineno (the first component of the tuple) is wrong. It should be either 1 or 2 (depending on whether you expect the lineno of gettext or the lineno of the value), but not 3.

This is because the code setting message_lineno is executed every time an opening parenthesis is encountered and funcname is set:
https://github.com/python-babel/babel/blob/f91754b01cb9f32b83aeaa80b74ed10b5dfccb6a/babel/messages/extract.py#L526-L533
while funcname is only changed if the name is in keywords (i.e., if the name is one of the gettext names):
https://github.com/python-babel/babel/blob/f91754b01cb9f32b83aeaa80b74ed10b5dfccb6a/babel/messages/extract.py#L615-L616

So, in the case where you have a gettext call -setting funcname- with another function call inside its arguments -resulting in an opening parenthesis-, you meet the two conditions to change the value of lineno for something that might be wrong, for example, if you spread the arguments to gettext over multiple lines

## Steps to Reproduce

Copy-paste this to your python REPL:
```py
from io import BytesIO
from babel.messages.extract import extract

file = b"""_(
    "hello",
    dummy_fn_call()
)"""

list(extract("python", BytesIO(file)))
```

## Actual Results

```py
[(3, 'hello', [], None)]
```

## Expected Results
```py
[(1, 'hello', [], None)]
```
or maybe
```py
[(2, 'hello', [], None)]
```
depending on whether you expect the lineno of gettext or the lineno of the value

## Additional Information
The first and last test cases of `test_comments_with_calls_that_spawn_multiple_lines` should fail, assuming you decide to consider the lineno to be the lineno of the gettext call. It currently fails if you replace the call to `len` in the test cases with something that isn't a function call


Discussion:
I also wrote a quick-and-dirty fix:
```diff
diff --git a/babel/messages/extract.py b/babel/messages/extract.py
index 48573ee..182a74b 100644
--- a/babel/messages/extract.py
+++ b/babel/messages/extract.py
@@ -502,6 +502,7 @@ def extract_python(
     :param options: a dictionary of additional options (optional)
     :rtype: ``iterator``
     """
+    last_name_token = ""
     funcname = lineno = message_lineno = None
     stack_depth = -1
     buf = []
@@ -521,6 +522,8 @@ def extract_python(
     current_fstring_start = None
 
     for tok, value, (lineno, _), _, _ in tokens:
+        if tok == NAME:
+            last_name_token = value
         if stack_depth == -1 and tok == NAME and value in ('def', 'class'):
             in_def = True
         elif tok == OP and value == '(':
@@ -530,7 +533,8 @@ def extract_python(
                 in_def = False
                 continue
             if funcname:
-                message_lineno = lineno
+                if last_name_token in keywords:
+                    message_lineno = lineno
                 stack_depth += 1
         elif in_def and tok == OP and value == ':':
             # End of a class definition without parens
diff --git a/tests/messages/test_extract.py b/tests/messages/test_extract.py
index 7d3a05a..852cdea 100644
--- a/tests/messages/test_extract.py
+++ b/tests/messages/test_extract.py
@@ -97,10 +97,10 @@ add_notice(req, ngettext("Bar deleted.",
         messages = list(extract.extract_python(buf, ('ngettext', '_'), ['NOTE:'],
 
                                                {'strip_comment_tags': False}))
-        assert messages[0] == (3, 'ngettext', ('Catalog deleted.', 'Catalogs deleted.', None), ['NOTE: This Comment SHOULD Be Extracted'])
+        assert messages[0] == (2, 'ngettext', ('Catalog deleted.', 'Catalogs deleted.', None), ['NOTE: This Comment SHOULD Be Extracted'])
         assert messages[1] == (6, '_', 'Locale deleted.', ['NOTE: This Comment SHOULD Be Extracted'])
         assert messages[2] == (10, 'ngettext', ('Foo deleted.', 'Foos deleted.', None), ['NOTE: This Comment SHOULD Be Extracted'])
-        assert messages[3] == (15, 'ngettext', ('Bar deleted.', 'Bars deleted.', None), ['NOTE: This Comment SHOULD Be Extracted', 'NOTE: And This One Too'])
+        assert messages[3] == (14, 'ngettext', ('Bar deleted.', 'Bars deleted.', None), ['NOTE: This Comment SHOULD Be Extracted', 'NOTE: And This One Too'])
 
     def test_declarations(self):
         buf = BytesIO(b"""\
```
Thanks for the investigation!

Just for reference, GNU's gettext sets the line number to 2:

```sh
> xgettext hello.py
> cat messages.po 
...

#: line.py:2
msgid "hello"
msgstr ""
```

I think it makes sense to try to be consistent with xgettext here, i.e. use lineno=2. Would you like to open a PR?


> I think it makes sense to try to be consistent with xgettext here, i.e. use lineno=2.
@tomasr8

Agreed, and in addition to being consistent with xgettext, it makes more sense to use the number of the line with the actual message.
It might be a bit tricky to implement, though: I imagine it'll require pairing each message with its lineno, which would change the current API and break the custom extract methods. What do you think?

> Would you like to open a PR?

I don't mind giving it a try, but I'm not sure I'll be able to free up enough time to look into it at the moment
> I imagine it'll require pairing each message with its lineno, which would change the current API and break the custom extract methods.

How would this break custom extractors?



Patch:
diff --git a/babel/messages/extract.py b/babel/messages/extract.py
index 8d4bbeaf8..1b2a37fc6 100644
--- a/babel/messages/extract.py
+++ b/babel/messages/extract.py
@@ -33,7 +33,7 @@
 from functools import lru_cache
 from os.path import relpath
 from textwrap import dedent
-from tokenize import COMMENT, NAME, OP, STRING, generate_tokens
+from tokenize import COMMENT, NAME, NL, OP, STRING, generate_tokens
 from typing import TYPE_CHECKING, Any
 
 from babel.messages._compat import find_entrypoints
@@ -530,7 +530,6 @@ def extract_python(
                 in_def = False
                 continue
             if funcname:
-                message_lineno = lineno
                 call_stack += 1
         elif in_def and tok == OP and value == ':':
             # End of a class definition without parens
@@ -580,11 +579,15 @@ def extract_python(
             elif tok == STRING:
                 val = _parse_python_string(value, encoding, future_flags)
                 if val is not None:
+                    if not message_lineno:
+                        message_lineno = lineno
                     buf.append(val)
 
             # Python 3.12+, see https://peps.python.org/pep-0701/#new-tokens
             elif tok == FSTRING_START:
                 current_fstring_start = value
+                if not message_lineno:
+                    message_lineno = lineno
             elif tok == FSTRING_MIDDLE:
                 if current_fstring_start is not None:
                     current_fstring_start += value
@@ -608,6 +611,9 @@ def extract_python(
                     # for the comment to still be a valid one
                     old_lineno, old_comment = translator_comments.pop()
                     translator_comments.append((old_lineno + 1, old_comment))
+
+            elif tok != NL and not message_lineno:
+                message_lineno = lineno
         elif call_stack > 0 and tok == OP and value == ')':
             call_stack -= 1
         elif funcname and call_stack == -1:


Test patch:
diff --git a/tests/messages/test_extract.py b/tests/messages/test_extract.py
index 7d3a05aa7..bcc6aa475 100644
--- a/tests/messages/test_extract.py
+++ b/tests/messages/test_extract.py
@@ -34,6 +34,11 @@ def test_nested_calls(self):
 msg8 = gettext('Rabbit')
 msg9 = dgettext('wiki', model.addPage())
 msg10 = dngettext(getDomain(), 'Page', 'Pages', 3)
+msg11 = ngettext(
+    "bunny",
+    "bunnies",
+    len(bunnies)
+)
 """)
         messages = list(extract.extract_python(buf,
                                                extract.DEFAULT_KEYWORDS.keys(),
@@ -49,6 +54,7 @@ def test_nested_calls(self):
             (8, 'gettext', 'Rabbit', []),
             (9, 'dgettext', ('wiki', None), []),
             (10, 'dngettext', (None, 'Page', 'Pages', None), []),
+            (12, 'ngettext', ('bunny', 'bunnies', None), []),
         ]
 
     def test_extract_default_encoding_ascii(self):
@@ -97,10 +103,10 @@ def test_comments_with_calls_that_spawn_multiple_lines(self):
         messages = list(extract.extract_python(buf, ('ngettext', '_'), ['NOTE:'],
 
                                                {'strip_comment_tags': False}))
-        assert messages[0] == (3, 'ngettext', ('Catalog deleted.', 'Catalogs deleted.', None), ['NOTE: This Comment SHOULD Be Extracted'])
+        assert messages[0] == (2, 'ngettext', ('Catalog deleted.', 'Catalogs deleted.', None), ['NOTE: This Comment SHOULD Be Extracted'])
         assert messages[1] == (6, '_', 'Locale deleted.', ['NOTE: This Comment SHOULD Be Extracted'])
         assert messages[2] == (10, 'ngettext', ('Foo deleted.', 'Foos deleted.', None), ['NOTE: This Comment SHOULD Be Extracted'])
-        assert messages[3] == (15, 'ngettext', ('Bar deleted.', 'Bars deleted.', None), ['NOTE: This Comment SHOULD Be Extracted', 'NOTE: And This One Too'])
+        assert messages[3] == (14, 'ngettext', ('Bar deleted.', 'Bars deleted.', None), ['NOTE: This Comment SHOULD Be Extracted', 'NOTE: And This One Too'])
 
     def test_declarations(self):
         buf = BytesIO(b"""\