Repository: shader-slang/slang
Pull request #6989: Parse char literals as integers
Author: login: juliusikkala
Labels: name: pr: non-breaking
description: PRs without breaking changes
Created: 2025-05-03T20:22:41Z | Merged: 2025-05-06T17:56:18Z

Closes #6984 and #6414. Conveniently, there was already a token type for character literals in the codebase, so only a conversion to an integer and small parser changes were needed to make this work. The approach taken here is to treat char literals as integer literals corresponding to the Unicode codepoint of the given char.

This PR leaves actual char literal AST / IR types as future work (not volunteering for that one, seems like it's quite a lot more work and is not immediately useful to me or the mentioned issues). They would be necessary to keep char literals as char literals on targets that support them, like HLSL and C++.

Linked issue #6984: Feature Request: support character literals like HLSL does
Hello!

When using slang to convert HLSL shaders to WGSL, slang complains when using character literals such as '1'.

Character literals are helpful when making debugging functionality, like printf from a shader.

HLSL is just fine with these character literals and converts them into their integral ascii value.

Could we have the same functionality in slang?

Thanks!
Linked issue comments: user: login: jarcherNV
created_at: 2025-05-02T17:17:52Z
body: Is it possible for you to share a simple example shader that shows the behavior you're looking for?
author_association: COLLABORATOR
user: login: fairywreath
created_at: 2025-05-02T17:22:22Z
body: Request is similar to #6414.
author_association: CONTRIBUTOR
user: login: Atrix256
created_at: 2025-05-02T17:47:52Z
body: With a test.slang of:
```
[numthreads(8, 8, 1)]
void main()
{
    uint offset = 'a';
    // I might use this value to calculate a location in a font texture where the letter 'a' begins;
}
```

I run the command:
`.\slangc .\test.slang -target wgsl -entry main -stage compute -o test.wgsl`

It gives the error:
```
.\test.slang(4): error 20002: syntax error.
    uint offset = 'a';
                  ^~~
```

It would be nice if slang treated that as the integral ascii value 97.
As many targets of slang don't support character literals, I would be happy for slang to replace that character literal with the integral ascii value 97 in the output.

Thanks!
author_association: NONE
user: login: juliusikkala
created_at: 2025-05-03T20:25:41Z
body: This feature is also useful to me, so I investigated how much work it would be to implement and ended up just implementing a basic version of it.
author_association: CONTRIBUTOR

Diff:
diff --git a/source/compiler-core/slang-lexer.cpp b/source/compiler-core/slang-lexer.cpp
index d19ea388d85..358a0ceffc1 100644
--- a/source/compiler-core/slang-lexer.cpp
+++ b/source/compiler-core/slang-lexer.cpp
@@ -6,6 +6,7 @@
 //
 
 #include "core/slang-char-encode.h"
+#include "core/slang-string-escape-util.h"
 #include "slang-core-diagnostics.h"
 #include "slang-name.h"
 #include "slang-source-loc.h"
@@ -829,17 +830,48 @@ FloatingPointLiteralValue getFloatingPointLiteralValue(
     return value;
 }
 
-static void _lexStringLiteralBody(Lexer* lexer, char quote)
+IntegerLiteralValue getCharLiteralValue(Token const& token)
 {
+    String unquotedContent = StringEscapeUtil::unquote('\'', token.getContent());
+    StringBuilder unescaped(4);
+    auto escapeHandler = StringEscapeUtil::getHandler(StringEscapeUtil::Style::Cpp);
+    escapeHandler->appendUnescaped(unquotedContent.getUnownedSlice(), unescaped);
+
+    char const* cursor = unescaped.getBuffer();
+
+    IntegerLiteralValue codepoint = getUnicodePointFromUTF8([&]() { return *cursor++; });
+    return codepoint;
+}
+
+static void _lexStringLiteralBody(Lexer* lexer, char quote, bool singleChar)
+{
+    int len = 0;
     for (;;)
     {
         int c = _peek(lexer);
         if (c == quote)
         {
+            if (singleChar && len == 0)
+            { // Empty char literal - size must be exactly 1.
+                if (auto sink = lexer->getDiagnosticSink())
+                {
+                    sink->diagnose(_getSourceLoc(lexer), LexerDiagnostics::illegalCharacterLiteral);
+                }
+            }
             _advance(lexer);
             return;
         }
 
+        len++;
+
+        if (singleChar && len == 2)
+        { // Char literal about to have more than 1 char.
+            if (auto sink = lexer->getDiagnosticSink())
+            {
+                sink->diagnose(_getSourceLoc(lexer), LexerDiagnostics::illegalCharacterLiteral);
+            }
+        }
+
         switch (c)
         {
         case kEOF:
@@ -1346,12 +1378,12 @@ static TokenType _lexTokenImpl(Lexer* lexer)
 
     case '\"':
         _advance(lexer);
-        _lexStringLiteralBody(lexer, '\"');
+        _lexStringLiteralBody(lexer, '\"', false);
         return TokenType::StringLiteral;
 
     case '\'':
         _advance(lexer);
-        _lexStringLiteralBody(lexer, '\'');
+        _lexStringLiteralBody(lexer, '\'', true);
         return TokenType::CharLiteral;
 
 
diff --git a/source/compiler-core/slang-lexer.h b/source/compiler-core/slang-lexer.h
index c39d130b748..99ad8768130 100644
--- a/source/compiler-core/slang-lexer.h
+++ b/source/compiler-core/slang-lexer.h
@@ -184,6 +184,8 @@ IntegerLiteralValue getIntegerLiteralValue(
 FloatingPointLiteralValue getFloatingPointLiteralValue(
     Token const& token,
     UnownedStringSlice* outSuffix = 0);
+
+IntegerLiteralValue getCharLiteralValue(Token const& token);
 } // namespace Slang
 
 #endif
diff --git a/source/slang/slang-parser.cpp b/source/slang/slang-parser.cpp
index c17a086a747..7ff0a6a2fad 100644
--- a/source/slang/slang-parser.cpp
+++ b/source/slang/slang-parser.cpp
@@ -7584,6 +7584,21 @@ static Expr* parseAtomicExpr(Parser* parser)
 
             return constExpr;
         }
+
+    case TokenType::CharLiteral:
+        {
+            IntegerLiteralExpr* constExpr = parser->astBuilder->create<IntegerLiteralExpr>();
+            parser->FillPosition(constExpr);
+
+            auto token = parser->tokenReader.advanceToken();
+            constExpr->token = token;
+
+            IntegerLiteralValue value = getCharLiteralValue(token);
+            constExpr->value = value;
+            constExpr->suffixType = BaseType::UInt;
+            return constExpr;
+        }
+
     case TokenType::CompletionRequest:
         {
             VarExpr* varExpr = parser->astBuilder->create<VarExpr>();
diff --git a/tests/front-end/char-literal.slang b/tests/front-end/char-literal.slang
new file mode 100644
index 00000000000..69d106c274d
--- /dev/null
+++ b/tests/front-end/char-literal.slang
@@ -0,0 +1,30 @@
+//TEST(compute):COMPARE_COMPUTE(filecheck-buffer=CHECK): -vk -compute -shaderobj
+//TEST(compute):COMPARE_COMPUTE(filecheck-buffer=CHECK): -dx12 -compute -shaderobj
+//TEST(compute):COMPARE_COMPUTE(filecheck-buffer=CHECK): -cpu -compute -shaderobj
+
+// Test that char literals work as expected.
+
+// CHECK: 48
+// CHECK-NEXT: 65
+// CHECK-NEXT: 6C
+// CHECK-NEXT: 6C
+// CHECK-NEXT: 6F
+// CHECK-NEXT: 4E16
+// CHECK-NEXT: 754C
+// CHECK-NEXT: A
+
+//TEST_INPUT:ubuffer(data=[0 0 0 0 0 0 0 0], stride=4):out,name outputBuffer
+RWStructuredBuffer<int> outputBuffer;
+
+[numthreads(1, 1, 1)]
+void computeMain()
+{
+    outputBuffer[0] = 'H';
+    outputBuffer[1] = 'e';
+    outputBuffer[2] = 'l';
+    outputBuffer[3] = 'l';
+    outputBuffer[4] = 'o';
+    outputBuffer[5] = '世';
+    outputBuffer[6] = '界';
+    outputBuffer[7] = '\n';
+}


Reviews:
user: login: csyonghe
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-03T23:13:40Z
user: login: csyonghe
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-03T23:15:30Z
user: login: juliusikkala
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-04T00:08:10Z
user: login: csyonghe
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-05T19:30:32Z
user: login: juliusikkala
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T20:16:54Z

Review comments:
diff_hunk: @@ -829,17 +829,47 @@ FloatingPointLiteralValue getFloatingPointLiteralValue(
     return value;
 }
 
-static void _lexStringLiteralBody(Lexer* lexer, char quote)
+IntegerLiteralValue getCharLiteralValue(Token const& token)
 {
+    const UnownedStringSlice content = token.getContent();
path: source/compiler-core/slang-lexer.cpp
user: login: csyonghe
body: what if the token content contains escape sequences?

e.g. `'\t'`?
created_at: 2025-05-03T23:13:40Z
author_association: COLLABORATOR
diff_hunk: @@ -829,17 +829,47 @@ FloatingPointLiteralValue getFloatingPointLiteralValue(
     return value;
 }
 
-static void _lexStringLiteralBody(Lexer* lexer, char quote)
+IntegerLiteralValue getCharLiteralValue(Token const& token)
 {
+    const UnownedStringSlice content = token.getContent();
path: source/compiler-core/slang-lexer.cpp
user: login: csyonghe
body: May want to use `StringEscapeUtil::unquote()` here.
created_at: 2025-05-03T23:15:25Z
author_association: COLLABORATOR
diff_hunk: @@ -829,17 +829,47 @@ FloatingPointLiteralValue getFloatingPointLiteralValue(
     return value;
 }
 
-static void _lexStringLiteralBody(Lexer* lexer, char quote)
+IntegerLiteralValue getCharLiteralValue(Token const& token)
 {
+    const UnownedStringSlice content = token.getContent();
path: source/compiler-core/slang-lexer.cpp
user: login: juliusikkala
body: That's a really good point. I've now added an escaped char in the test and unquoted appropriately. It had to be done in a bit of a roundabout way; `StringEscapeUtil::unquote()` doesn't unescape, and `StringEscapeUtil::unescapeString` assumes that the string starts with `"` and not `'`.
created_at: 2025-05-04T00:08:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -829,17 +830,48 @@ FloatingPointLiteralValue getFloatingPointLiteralValue(
     return value;
 }
 
-static void _lexStringLiteralBody(Lexer* lexer, char quote)
+IntegerLiteralValue getCharLiteralValue(Token const& token)
 {
+    String unquotedContent = StringEscapeUtil::unquote('\'', token.getContent());
+    StringBuilder unescaped(4);
+    auto escapeHandler = StringEscapeUtil::getHandler(StringEscapeUtil::Style::Cpp);
+    escapeHandler->appendUnescaped(unquotedContent.getUnownedSlice(), unescaped);
path: source/compiler-core/slang-lexer.cpp
user: login: csyonghe
body: can just call `appendUnquoted` which calls `unquote + unescape` inside.
created_at: 2025-05-05T19:30:18Z
author_association: COLLABORATOR
diff_hunk: @@ -829,17 +830,48 @@ FloatingPointLiteralValue getFloatingPointLiteralValue(
     return value;
 }
 
-static void _lexStringLiteralBody(Lexer* lexer, char quote)
+IntegerLiteralValue getCharLiteralValue(Token const& token)
 {
+    String unquotedContent = StringEscapeUtil::unquote('\'', token.getContent());
+    StringBuilder unescaped(4);
+    auto escapeHandler = StringEscapeUtil::getHandler(StringEscapeUtil::Style::Cpp);
+    escapeHandler->appendUnescaped(unquotedContent.getUnownedSlice(), unescaped);
path: source/compiler-core/slang-lexer.cpp
user: login: juliusikkala
body: To get `appendUnquoted` to unquote `'` correctly, I'd need to create a custom `StringEscapeHandler` with `quoteChar = '\'';`, because it's `'"'` by default and refuses to unquote chars because of that. I can do that, but it just seems like a lot more code for little value.
created_at: 2025-05-05T20:16:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: csyonghe
created_at: 2025-05-03T23:07:21Z
body: Yeah, we still need to decide what `char` type in Slang should be. I am leaning towards that `char` should be a 32bit unicode code point, and String.operator[] should be returning the decoded unicode char. But this could be mean the operation is more heavyweight than what people would expect. So we may not have a type named exactly as `char` to avoid confusion with C++'s `char`, which isn't really a char.
author_association: COLLABORATOR
user: login: juliusikkala
created_at: 2025-05-04T17:51:32Z
body: I also agree that chars should definitely be 32-bit unicode codepoints. `char` carries a lot of C baggage. One option would be to just not have a 'char' type, and instead just treat char literals as `uint32_t` (which is what this PR does). At least that would be pretty explicit about the size, but wouldn't help at all with preserving literals into target languages. A name like `char32_t` could be nice.

Making `String.operator[]` decode UTF-8 to UTF-32 seems like a bad idea. For random access to a char in the middle of the string, you'd need to resolve the sizes of all preceding chars. So iterating over a string would suddenly become an O(n^2) operation, unless the string stores some lookup structure, but at that point, why not just make the String UTF-32 in the first place? So perhaps it'd make more sense to have some more controlled way of iterating over the string, like

```slang
uint32_t offset = 0;
while(let c = str.advance(offset))
{
}
```

Or just make Strings actually store chars in UTF-32.
author_association: CONTRIBUTOR
user: login: csyonghe
created_at: 2025-05-04T17:57:58Z
body: Another way is to not provide String.operator[], but to provide String.rawBytes that is a reference to uint8_t[].

We could provide sequential iterator to String that allows serial access to decoded characters.

We can also provide a .toUtf32String() that returns Utf32String() which allows direct operator[].
author_association: COLLABORATOR
user: login: csyonghe
created_at: 2025-05-04T18:01:00Z
body: That being said, we also need a language-wise convention for iterators. This quickly gets into lifetime/ownership problems which we don’t have a proper language spec for yet…
author_association: COLLABORATOR