Repository: edgedb/edgedb

Issue:
Extra features for the numeric constants
Tho things quite common in modern languages:
1. `123.` for floats (i.e. no digits in decimal part)
2. `1_000_000` for long numbers (underscores)

I'm going to implement it in rust tokenizer since the start if no one objects.



Discussion:
> I'm going to implement it in rust tokenizer since the start if no one objects.

Let's consider this **after** alpha 2, please.  We're all for implementing `1_000_000` -- this was on our mental to-do list.  As for `123.` -- need to check if we have no potential issues with it in our grammar.
> 1. `123.` for floats (i.e. no digits in decimal part)

We have the following allowed expression for accessing nested tuple values:
```
db> WITH a := (1,(2,3)) # could be any nested tuple
... SELECT a.1.1;
{3}
```
The problem is that `1.1` is tokenized by the lexer as a float literal and then there's a parser hack that correctly interprets it as a path instead. This hack (see [here](https://github.com/edgedb/edgedb/blob/eef5085bdbb20e286cc1ee4a0d38c37b54e07b8f/edb/edgeql/parser/grammar/expressions.py#L1009)) depends on knowing that the only valid float for tuple access is of the form `123.456`. Allowing a trailing `.` would complicate the grammar significantly. This begs the question whether the convenience of not typing a `0` is worth the extra complexity. Also, if we allow `1.` should we also allow `.1` as a valid number? Again, this will increase the parser complexity significantly as well.



> The problem is that 1.1 is tokenized by the lexer as a float literal and then there's a parser hack that correctly interprets it as a path instead. 

Well, how common is such access to nested tuples? Rust have the same problem and they decided to forbid it:
```
2 |     println!("{}", (1,).0.2.3)
  |                    -----^^^
  |                    |    |
  |                    |    unexpected token
  |                    help: try parenthesizing the first index: `((1,).0).2`
```
While I'm okay with forbidding `0.` for now. I don't think this parser hack is a good idea in the long term.
Why? It works and makes tuple access syntax obvious for users. Basically paths are `Object.link.property`, and for tuples we have `namedtup.field.another_field` or `tup.0.1`. On the contrary, `1.` isn't a common way of specifying floats, I rarely see when I'm browsing code in any language.
Another thought: we can always parse `<dot><digits>` as `<dot><integer><dot><integer>` and never as float. This introduces a little bit of state to tokenizer, but the state in tokenizer is quite common pattern.

And this doesn't prohibit `0.`

> Let's consider this after alpha 2, please.

Sure, rust parser isn't going to be integrated before we cut alpha 2.
> Another thought: we can always parse `<dot><digits>` as `<dot><integer><dot><integer>`

Like [this](https://github.com/edgedb/edgedb-rust/commit/ee8c555423c31e813fa0d2035f8dc4cabe483cbf) (that last error message could be clearer, though) 
> Like this (that last error message could be clearer, though)

Yeah, sure. It should work if we can add a few relatively simple hacks to the lexer to make both `0.` and `.0` **and** tuple paths without parens!
You're forgetting `1.e123` (perfectly fine in JS and Python). What this all means is that there would be several distinct lexer tokens to account for different options and then also messy parser rules (because half of them would have to account for rules terminating in a `DOT`, while the other half would have a `DOT` in the middle). Oh, and then there's also the duplication of some of these rules for `decimal`, like `1.n` and `1.e123n`. The `123.` token would also have to be accounted for in several more rules, the ones dealing with `.<` and `.>`. 

In short, the changes are not as trivial and obvious as they seem and In the end effect for the user is minimal. 

They would introduce backwards-compatible syntax, if they are needed later, though.

(Actually, the remark about the `.>` made me realize that currently this form of the operator is not working properly with integer-indexed tuples, but works with named tuples.)
Current rust tokenizer ([added a test to make sure](https://github.com/edgedb/edgedb-rust/commit/98c32940aabfab548b9a4b77433b1898a1c18f80)) parses as follows:
1. `tuple.1.<` parses fine as `tuple . 1  .<`
2. `1.<` parses as `1. <` but `.<` doesn't make sense on a number as far as I understand
3. similarly with `1.e123` is just a float, but...
4. `xxx.1.e123` is `xxx . 1 . e123` (i.e. tuple access, attribute access)

I'm not sure if (2) might cause confusion for somebody. Fortunately, we don't need to decide it until alpha 2 :)
> `.<` doesn't make sense on a number as far as I understand

You're right, I overhastily included that as valid, but `.>` is supposed to be perfectly equivalent to `.` and valid anywhere where `.` navigation/dereferencing is valid.

@tailhook @vpetrovykh Anything left to do in this one?
Yes #1416

Resolving patch:
diff --git a/edb/edgeql/compiler/expr.py b/edb/edgeql/compiler/expr.py
--- a/edb/edgeql/compiler/expr.py
+++ b/edb/edgeql/compiler/expr.py
@@ -176,7 +176,8 @@ def compile_BaseConstant(
         std_type = 'std::str'
         node_cls = irast.StringConstant
     elif isinstance(expr, qlast.IntegerConstant):
-        int_value = int(expr.value)
+        value = value.replace("_", "")
+        int_value = int(value)
         if expr.is_negative:
             int_value = -int_value
             value = f'-{value}'
@@ -184,20 +185,21 @@ def compile_BaseConstant(
         std_type = 'std::int64'
         node_cls = irast.IntegerConstant
     elif isinstance(expr, qlast.FloatConstant):
+        value = value.replace("_", "")
         if expr.is_negative:
             value = f'-{value}'
         std_type = 'std::float64'
         node_cls = irast.FloatConstant
     elif isinstance(expr, qlast.DecimalConstant):
         assert value[-1] == 'n'
-        value = value[:-1]
+        value = value[:-1].replace("_", "")
         if expr.is_negative:
             value = f'-{value}'
         std_type = 'std::decimal'
         node_cls = irast.DecimalConstant
     elif isinstance(expr, qlast.BigintConstant):
         assert value[-1] == 'n'
-        value = value[:-1]
+        value = value[:-1].replace("_", "")
         if expr.is_negative:
             value = f'-{value}'
         std_type = 'std::bigint'


Test patch:
diff --git a/edb/edgeql-parser/tests/tokenizer.rs b/edb/edgeql-parser/tests/tokenizer.rs
--- a/edb/edgeql-parser/tests/tokenizer.rs
+++ b/edb/edgeql-parser/tests/tokenizer.rs
@@ -59,6 +59,8 @@ fn idents() {
     assert_eq!(tok_err(" + __test__"),
         "Unexpected `identifiers surrounded by double underscores \
         are forbidden`");
+    assert_eq!(tok_str("_1024"), ["_1024"]);
+    assert_eq!(tok_typ("_1024"), [Ident]);
 }
 
 #[test]
@@ -173,10 +175,6 @@ fn dot_tokens() {
 fn tuple_dot_vs_float() {
     assert_eq!(tok_str("tuple.1.<"), ["tuple", ".", "1", ".<"]);
     assert_eq!(tok_typ("tuple.1.<"), [Ident, Dot, IntConst, BackwardLink]);
-    assert_eq!(tok_str("1.<"), ["1.", "<"]);
-    assert_eq!(tok_typ("1.<"), [FloatConst, Less]);
-    assert_eq!(tok_str("1.e123"), ["1.e123"]);
-    assert_eq!(tok_typ("1.e123"), [FloatConst]);
     assert_eq!(tok_str("tuple.1.e123"), ["tuple", ".", "1", ".", "e123"]);
     assert_eq!(tok_typ("tuple.1.e123"), [Ident, Dot, IntConst, Dot, Ident]);
 }
@@ -216,14 +214,19 @@ fn integer() {
     assert_eq!(tok_typ("*0"), [Mul, IntConst]);
     assert_eq!(tok_str("123"), ["123"]);
     assert_eq!(tok_typ("123"), [IntConst]);
+    assert_eq!(tok_str("123_"), ["123_"]);
+    assert_eq!(tok_typ("123_"), [IntConst]);
+    assert_eq!(tok_str("123_456"), ["123_456"]);
+    assert_eq!(tok_typ("123_456"), [IntConst]);
 
     assert_eq!(tok_str("0 "), ["0"]);
     assert_eq!(tok_typ("0 "), [IntConst]);
     assert_eq!(tok_str("123 "), ["123"]);
     assert_eq!(tok_typ("123 "), [IntConst]);
-
-    assert_eq!(tok_err("01"),
-        "Unexpected `leading zeros are not allowed in numbers`");
+    assert_eq!(tok_str("123_ "), ["123_"]);
+    assert_eq!(tok_typ("123_ "), [IntConst]);
+    assert_eq!(tok_str("123_456 "), ["123_456"]);
+    assert_eq!(tok_typ("123_456 "), [IntConst]);
 }
 
 #[test]
@@ -238,6 +241,10 @@ fn bigint() {
     assert_eq!(tok_typ("123e3n"), [BigIntConst]);
     assert_eq!(tok_str("123e+99n"), ["123e+99n"]);
     assert_eq!(tok_typ("123e+99n"), [BigIntConst]);
+    assert_eq!(tok_str("123_n"), ["123_n"]);
+    assert_eq!(tok_typ("123_n"), [BigIntConst]);
+    assert_eq!(tok_str("123_456n"), ["123_456n"]);
+    assert_eq!(tok_typ("123_456n"), [BigIntConst]);
 
     assert_eq!(tok_str("0n "), ["0n"]);
     assert_eq!(tok_typ("0n "), [BigIntConst]);
@@ -247,14 +254,14 @@ fn bigint() {
     assert_eq!(tok_typ("123e3n "), [BigIntConst]);
     assert_eq!(tok_str("123e+99n "), ["123e+99n"]);
     assert_eq!(tok_typ("123e+99n "), [BigIntConst]);
-    assert_eq!(tok_err("01n"),
-        "Unexpected `leading zeros are not allowed in numbers`");
+    assert_eq!(tok_str("123_n "), ["123_n"]);
+    assert_eq!(tok_typ("123_n "), [BigIntConst]);
+    assert_eq!(tok_str("123_456n "), ["123_456n"]);
+    assert_eq!(tok_typ("123_456n "), [BigIntConst]);
 }
 
 #[test]
 fn float() {
-    assert_eq!(tok_str("0."), ["0."]);
-    assert_eq!(tok_typ("0."), [FloatConst]);
     assert_eq!(tok_str("     0.0"), ["0.0"]);
     assert_eq!(tok_typ("     0.0"), [FloatConst]);
     assert_eq!(tok_str("123.999"), ["123.999"]);
@@ -269,11 +276,19 @@ fn float() {
     assert_eq!(tok_typ("123e3"), [FloatConst]);
     assert_eq!(tok_str("123e+99"), ["123e+99"]);
     assert_eq!(tok_typ("123e+99"), [FloatConst]);
+    assert_eq!(tok_str("123e+99_"), ["123e+99_"]);
+    assert_eq!(tok_typ("123e+99_"), [FloatConst]);
+    assert_eq!(tok_str("123e+9_9"), ["123e+9_9"]);
+    assert_eq!(tok_typ("123e+9_9"), [FloatConst]);
     assert_eq!(tok_str("2345e-7"), ["2345e-7"]);
     assert_eq!(tok_typ("2345e-7"), [FloatConst]);
+    assert_eq!(tok_str("2_345e-7"), ["2_345e-7"]);
+    assert_eq!(tok_typ("2_345e-7"), [FloatConst]);
+    assert_eq!(tok_str("1_023.9_099"), ["1_023.9_099"]);
+    assert_eq!(tok_typ("1_023.9_099"), [FloatConst]);
+    assert_eq!(tok_str("1_023_.9_099_"), ["1_023_.9_099_"]);
+    assert_eq!(tok_typ("1_023_.9_099_"), [FloatConst]);
 
-    assert_eq!(tok_str("0. "), ["0."]);
-    assert_eq!(tok_typ("0. "), [FloatConst]);
     assert_eq!(tok_str("     0.0 "), ["0.0"]);
     assert_eq!(tok_typ("     0.0 "), [FloatConst]);
     assert_eq!(tok_str("123.999 "), ["123.999"]);
@@ -288,8 +303,12 @@ fn float() {
     assert_eq!(tok_typ("123e3 "), [FloatConst]);
     assert_eq!(tok_str("123e+99 "), ["123e+99"]);
     assert_eq!(tok_typ("123e+99 "), [FloatConst]);
+    assert_eq!(tok_str("123e+99_ "), ["123e+99_"]);
+    assert_eq!(tok_typ("123e+99_ "), [FloatConst]);
     assert_eq!(tok_str("2345e-7 "), ["2345e-7"]);
     assert_eq!(tok_typ("2345e-7 "), [FloatConst]);
+    assert_eq!(tok_str("1_023_.9_099_ "), ["1_023_.9_099_"]);
+    assert_eq!(tok_typ("1_023_.9_099_ "), [FloatConst]);
 
     assert_eq!(tok_err("01.2"),
         "Unexpected `leading zeros are not allowed in numbers`");
@@ -297,8 +316,6 @@ fn float() {
 
 #[test]
 fn decimal() {
-    assert_eq!(tok_str("0.n"), ["0.n"]);
-    assert_eq!(tok_typ("0.n"), [DecimalConst]);
     assert_eq!(tok_str("     0.0n"), ["0.0n"]);
     assert_eq!(tok_typ("     0.0n"), [DecimalConst]);
     assert_eq!(tok_str("123.999n"), ["123.999n"]);
@@ -311,9 +328,17 @@ fn decimal() {
     assert_eq!(tok_typ("2345.567e-7n"), [DecimalConst]);
     assert_eq!(tok_str("2345e-7n"), ["2345e-7n"]);
     assert_eq!(tok_typ("2345e-7n"), [DecimalConst]);
+    assert_eq!(tok_str("2_345e-7n"), ["2_345e-7n"]);
+    assert_eq!(tok_typ("2_345e-7n"), [DecimalConst]);
+    assert_eq!(tok_str("1_023.9_099n"), ["1_023.9_099n"]);
+    assert_eq!(tok_typ("1_023.9_099n"), [DecimalConst]);
+    assert_eq!(tok_str("1_023_.9_099_n"), ["1_023_.9_099_n"]);
+    assert_eq!(tok_typ("1_023_.9_099_n"), [DecimalConst]);
+    assert_eq!(tok_str("2_345e-7n"), ["2_345e-7n"]);
+    assert_eq!(tok_typ("2_345e-7n"), [DecimalConst]);
+    assert_eq!(tok_str("2_345e-7_7n"), ["2_345e-7_7n"]);
+    assert_eq!(tok_typ("2_345e-7_7n"), [DecimalConst]);
 
-    assert_eq!(tok_str("0.n "), ["0.n"]);
-    assert_eq!(tok_typ("0.n "), [DecimalConst]);
     assert_eq!(tok_str("     0.0n "), ["0.0n"]);
     assert_eq!(tok_typ("     0.0n "), [DecimalConst]);
     assert_eq!(tok_str("123.999n "), ["123.999n"]);
@@ -423,6 +448,18 @@ fn numbers_from_py() {
 
 #[test]
 fn num_errors() {
+    assert_eq!(tok_err("0. "),
+        "Expected `expected digit after dot, found end of decimal`");
+    assert_eq!(tok_err("1.<"),
+        "Expected `expected digit after dot, found end of decimal`");
+    assert_eq!(tok_err("0.n"),
+        "Expected `expected digit after dot, found suffix`");
+    assert_eq!(tok_err("0.e1"),
+        "Expected `expected digit after dot, found exponent`");
+    assert_eq!(tok_err("0.e1n"),
+        "Expected `expected digit after dot, found exponent`");
+    assert_eq!(tok_err("0."),
+        "Expected `expected digit after dot, found end of decimal`");
     assert_eq!(tok_err("1.0.x"),
         "Unexpected `extra decimal dot in number`");
     assert_eq!(tok_err("1.0e1."),
@@ -442,6 +479,15 @@ fn num_errors() {
     assert_eq!(tok_err("1.0e "),
         "Unexpected `optional `+` or `-` \
         followed by digits must follow `e` in float const`");
+    assert_eq!(tok_err("1.0e_"),
+        "Unexpected `optional `+` or `-` \
+        followed by digits must follow `e` in float const`");
+    assert_eq!(tok_err("1.0e_ "),
+        "Unexpected `optional `+` or `-` \
+        followed by digits must follow `e` in float const`");
+    assert_eq!(tok_err("1.0e_1"),
+        "Unexpected `optional `+` or `-` \
+        followed by digits must follow `e` in float const`");
     assert_eq!(tok_err("1.0e+"),
         "Unexpected `optional `+` or `-` \
         followed by digits must follow `e` in float const`");
@@ -463,6 +509,16 @@ fn num_errors() {
     assert_eq!(tok_err("100O00"),
         "Unexpected `suffix \"O00\" is invalid for numbers, \
         perhaps mixed up letter `O` with zero `0`?`");
+    assert_eq!(tok_err("01"),
+        "Unexpected `leading zeros are not allowed in numbers`");
+    assert_eq!(tok_err("01n"),
+        "Unexpected `leading zeros are not allowed in numbers`");
+    assert_eq!(tok_err("01_n"),
+        "Unexpected `leading zeros are not allowed in numbers`");
+    assert_eq!(tok_err("0_1_n"),
+        "Unexpected `leading zeros are not allowed in numbers`");
+    assert_eq!(tok_err("0_1n"),
+        "Unexpected `leading zeros are not allowed in numbers`");
 }
 
 #[test]
diff --git a/tests/test_edgeql_ddl.py b/tests/test_edgeql_ddl.py
--- a/tests/test_edgeql_ddl.py
+++ b/tests/test_edgeql_ddl.py
@@ -16,7 +16,6 @@
 # limitations under the License.
 #
 
-
 import edgedb
 
 from edb.testbase import server as tb
diff --git a/tests/test_edgeql_expressions.py b/tests/test_edgeql_expressions.py
--- a/tests/test_edgeql_expressions.py
+++ b/tests/test_edgeql_expressions.py
@@ -380,6 +380,11 @@ async def test_edgeql_expr_literals_02(self):
                     r'''SELECT <int64>'3689348814741900000000000' ''',
                 )
 
+        with self.assertRaisesRegex(edgedb.EdgeQLSyntaxError,
+                                    'expected digit after dot'):
+            async with self.con.transaction():
+                await self.con.fetchone('SELECT 0. ')
+
     async def test_edgeql_expr_op_02(self):
         await self.assert_query_result(
             r'''SELECT 40 ^ 2;''',
diff --git a/tests/test_edgeql_syntax.py b/tests/test_edgeql_syntax.py
--- a/tests/test_edgeql_syntax.py
+++ b/tests/test_edgeql_syntax.py
@@ -17,6 +17,7 @@
 #
 
 
+import decimal
 import re
 import unittest  # NOQA
 
@@ -109,6 +110,7 @@ def test_edgeql_syntax_constants_01(self):
         SELECT +7;
         SELECT -7;
         SELECT 551;
+        SELECT 1_024;
         """
 
     def test_edgeql_syntax_constants_02(self):
@@ -145,7 +147,9 @@ def test_edgeql_syntax_constants_04(self):
         SELECT 3.5432e20;
         SELECT 3.5432e+20;
         SELECT 3.5432e-20;
+        SELECT 3.543_2e-20;
         SELECT 354.32e-20;
+        SELECT 2_354.32e-20;
 
 % OK %
 
@@ -155,7 +159,9 @@ def test_edgeql_syntax_constants_04(self):
         SELECT 3.5432e20;
         SELECT 3.5432e+20;
         SELECT 3.5432e-20;
+        SELECT 3.543_2e-20;
         SELECT 354.32e-20;
+        SELECT 2_354.32e-20;
         """
 
     def test_edgeql_syntax_constants_05(self):
@@ -181,6 +187,8 @@ def test_edgeql_syntax_constants_07(self):
         SELECT 02;
         """
 
+    @tb.must_fail(errors.EdgeQLSyntaxError, 'expected digit after dot',
+                  line=2, col=16)
     def test_edgeql_syntax_constants_08(self):
         """
         SELECT 1.;
@@ -449,12 +457,15 @@ def test_edgeql_syntax_constants_43(self):
         SELECT -1n;
         SELECT 100000n;
         SELECT -100000n;
+        SELECT 100_000n;
+        SELECT -100_000n;
         SELECT -354.32n;
         SELECT 35400000000000.32n;
         SELECT -35400000000000000000.32n;
         SELECT 3.5432e20n;
         SELECT -3.5432e+20n;
         SELECT 3.5432e-20n;
+        SELECT 3.543_2e-20n;
         SELECT 354.32e-20n;
 
 % OK %
@@ -465,12 +476,15 @@ def test_edgeql_syntax_constants_43(self):
         SELECT -1n;
         SELECT 100000n;
         SELECT -100000n;
+        SELECT 100_000n;
+        SELECT -100_000n;
         SELECT -354.32n;
         SELECT 35400000000000.32n;
         SELECT -35400000000000000000.32n;
         SELECT 3.5432e20n;
         SELECT -3.5432e+20n;
         SELECT 3.5432e-20n;
+        SELECT 3.543_2e-20n;
         SELECT 354.32e-20n;
         """
 
@@ -3460,6 +3474,54 @@ def test_edgeql_syntax_ddl_function_48(self):
             std::int64 USING SQL FUNCTION 'aaa';
         """
 
+    async def test_edgeql_syntax_ddl_function_49(self):
+        # This test checks constants, but we have to do DDLs to test them
+        # with constant extraction disabled
+        await self.con.execute('''
+            CREATE FUNCTION test::constant_int() -> std::int64 {
+                USING (SELECT 1_024);
+            };
+            CREATE FUNCTION test::constant_bigint() -> std::bigint {
+                USING (SELECT 1_024n);
+            };
+            CREATE FUNCTION test::constant_float() -> std::float64 {
+                USING (SELECT 1_024.1_250);
+            };
+            CREATE FUNCTION test::constant_decimal() -> std::decimal {
+                USING (SELECT 1_024.1_024n);
+            };
+        ''')
+        try:
+            await self.assert_query_result(
+                r'''
+                    SELECT (
+                        int := test::constant_int(),
+                        bigint := test::constant_bigint(),
+                        float := test::constant_float(),
+                        decimal := test::constant_decimal(),
+                    )
+                ''',
+                [{
+                    "int": 1024,
+                    "bigint": 1024,
+                    "float": 1024.125,
+                    "decimal": 1024.1024,
+                }],
+                [{
+                    "int": 1024,
+                    "bigint": 1024,
+                    "float": 1024.125,
+                    "decimal": decimal.Decimal('1024.1024'),
+                }],
+            )
+        finally:
+            await self.con.execute("""
+                DROP FUNCTION test::constant_int();
+                DROP FUNCTION test::constant_float();
+                DROP FUNCTION test::constant_bigint();
+                DROP FUNCTION test::constant_decimal();
+            """)
+
     def test_edgeql_syntax_ddl_property_01(self):
         """
         CREATE ABSTRACT PROPERTY std::property {