Repository: tweag/nickel
Pull request #994 (2022-12-14 15:40:22)

Symbolic strings
This PR introduces a simple form of "symbolic strings" (#948), whereby the syntax `s%"..."%` is just de-sugared to an array.

matthew-healy: @yannham

> The first question is: should we have symbolic strings be multiline by default, or offer a separate version for multiline strings?

I think starting off by only supporting multiline symbolic strings makes sense. Especially as it's possible to just write mutliline symbolic strings on a single line. 

I can't currently think of a use case for a syntax like `s"..."` that isn't already supported by the multiline version, but maybe I'm missing something?

>The second question, which is independent, is what to do with interpolated blocks inside mutliline strings.

In the current iteration I just ignore the `indent` parameter. My thinking on this was that symbolic strings will always go through post-processing before they ever end up as regular strings, so making assumptions about how the interpolated values should be indented feels like it might be unnecessary, as Nickel has absolutely no control over the ultimate string value (assuming one is ever created).

To look at your example from the comment above:

```nickel
let bar = "alpha\nbeta" in
m%"
  foo
     %{bar}
  baz
"%
```

currently evaluates to

```
[ "
foo
  ", 
"alpha
beta",
"
baz"
]
```

This _feels_ like how I'd expect this to behave (though obviously that's completely unscientific :sweat_smile:) - it seems to make sense to give the exact value that was interpolated here. Though I suppose one place it could matter is if the symbolic string is intended to be evaluated into a language with meaningful whitespace... :thinking: 

Ultimately, I don't have particularly strong feelings & am happy with whatever you think makes sense. I do think what we have now is probably "good enough" for a first pass - what do you think about opening a separate issue for how to handle multiline interpolated values?
vkleen: I think @matthew-healy's example evaluation is the most reasonable. I wouldn't expect Nickel to process a string that's interpolated into a symbolic string at all. That's also in line with how normal string interpolation in indented multiline strings behaves.
yannham: >  That's also in line with how normal string interpolation in indented multiline strings behaves.

Precisely, I don't think this is the case:

```
nickel> let bar = "alpha\nbeta" in
m%"
  foo
     %{bar}
  baz
"%
"foo
   alpha
   beta
baz"

nickel> let bar = "alpha\nbeta" in
s%"
  foo
     %{bar}
  baz
"% |> array.foldl (++) ""
"foo
   alpha
beta
baz"

```

We carry explicitly the whole indentation of an interpolation (here e.g. 4 spaces for the `%{bar}` block) and re-apply that at runtime, once we've evalauted bar, to the beginning of each line. We can't do that, at least not trivially, for symbolic string, because the parsing function is the one reconstructing the string (if ever done) at the end. The best we can do is to pass this indentation. Maybe we don't care :man_shrugging: but I just wanted to raise the difference in behavior.  

> Ultimately, I don't have particularly strong feelings & am happy with whatever you think makes sense. I do think what we have now is probably "good enough" for a first pass - what do you think about opening a separate issue for how to handle multiline interpolated values?

Sure, this is the way to go.
vkleen: @yannham That's interesting! I tested with the following in `test.ncl`:
```nickel
let s = "a\nb" in
  m%"foo
  in %{s}
  bar"%
```
and `nickel -f test.ncl` results in
```
"foo
  in a
b
  bar"
```
I think the indent behaviour only kicks in if the multiline string starts with whitespace?

Patch:
diff --git a/lsp/nls/src/linearization/completed.rs b/lsp/nls/src/linearization/completed.rs
index 2582ee10..f65df0d4 100644
--- a/lsp/nls/src/linearization/completed.rs
+++ b/lsp/nls/src/linearization/completed.rs
@@ -127,13 +127,14 @@ impl Completed {
             _ => item,
         };
 
-        if let Some(MetaValue {
-            ref doc,
-            ref types,
-            ref contracts,
-            priority,
-            ..
-        }) = item.meta.as_ref()
+        if let Some(
+            meta @ MetaValue {
+                ref doc,
+                ref types,
+                priority,
+                ..
+            },
+        ) = item.meta.as_ref()
         {
             if let Some(doc) = doc {
                 extra.push(doc.to_owned());
@@ -141,14 +142,8 @@ impl Completed {
             if let Some(types) = types {
                 extra.push(types.label.tag.to_string());
             }
-            if !contracts.is_empty() {
-                extra.push(
-                    contracts
-                        .iter()
-                        .map(|contract| format!("{}", contract.label.types,))
-                        .collect::<Vec<_>>()
-                        .join(","),
-                );
+            if let Some(contracts) = meta.contracts_to_string() {
+                extra.push(contracts);
             }
 
             extra.push(format!("Merge Priority: {:?}", priority));
diff --git a/lsp/nls/src/requests/completion.rs b/lsp/nls/src/requests/completion.rs
index 768e0aa3..31bc4383 100644
--- a/lsp/nls/src/requests/completion.rs
+++ b/lsp/nls/src/requests/completion.rs
@@ -3,7 +3,9 @@ use codespan_lsp::position_to_byte_index;
 use lazy_static::lazy_static;
 use log::debug;
 use lsp_server::{RequestId, Response, ResponseError};
-use lsp_types::{CompletionItem, CompletionParams};
+use lsp_types::{
+    CompletionItem, CompletionItemKind, CompletionParams, Documentation, MarkupContent, MarkupKind,
+};
 use nickel_lang::{
     identifier::Ident,
     term::{MetaValue, RichTerm, Term},
@@ -29,18 +31,112 @@ use crate::{
 // We follow the path by traversing a term, type or contract which represents a record
 // and stop when there is nothing else on the path
 
+#[derive(Debug)]
+struct IdentWithType {
+    ident: Ident,
+    ty: Types,
+    item: Option<LinearizationItem<Types>>,
+}
+
+impl From<Ident> for IdentWithType {
+    fn from(ident: Ident) -> Self {
+        IdentWithType {
+            ident,
+            ty: Types(TypeF::Dyn),
+            item: None,
+        }
+    }
+}
+
+impl From<&str> for IdentWithType {
+    fn from(ident: &str) -> Self {
+        IdentWithType {
+            ident: Ident::from(ident),
+            ty: Types(TypeF::Dyn),
+            item: None,
+        }
+    }
+}
+
+impl IdentWithType {
+    fn detail(&self) -> String {
+        self.item
+            .as_ref()
+            .and_then(|item| {
+                item.meta.as_ref().and_then(|meta| {
+                    meta.types
+                        .as_ref()
+                        .map(|ty| ty.types.to_string())
+                        .or_else(|| meta.contracts_to_string())
+                })
+            })
+            .unwrap_or_else(|| self.ty.to_string())
+    }
+
+    fn completion_item_kind(&self) -> CompletionItemKind {
+        match &self.ty {
+            ty if ty.is_function_type() => CompletionItemKind::Function,
+            _ => CompletionItemKind::Property,
+        }
+    }
+
+    fn to_lsp_completion_item(&self) -> CompletionItem {
+        /// Attach quotes to a non-ASCII string
+        fn adjust_name(name: &str) -> String {
+            if name.is_ascii() {
+                String::from(name)
+            } else {
+                format!("\"{}\"", name)
+            }
+        }
+        let doc = || {
+            let item = self.item.as_ref()?;
+            let meta = item.meta.as_ref()?;
+            let doc = meta.doc.as_ref()?;
+            let doc = Documentation::MarkupContent(MarkupContent {
+                kind: MarkupKind::Markdown,
+                value: format!("{}\nMerge Priority {}", doc.clone(), meta.priority),
+            });
+            Some(doc)
+        };
+        CompletionItem {
+            label: adjust_name(self.ident.label()),
+            detail: Some(self.detail()),
+            kind: Some(self.completion_item_kind()),
+            documentation: doc(),
+            ..Default::default()
+        }
+    }
+}
+
 /// Find the record field associated with a particular ID in the linearization
 /// using lexical scoping rules.
 fn find_fields_from_term_kind(
     linearization: &Completed,
     id: ItemId,
     path: &mut Vec<Ident>,
-) -> Option<Vec<Ident>> {
+) -> Option<Vec<IdentWithType>> {
     let item = linearization.get_item(id)?;
     match item.kind {
         TermKind::Record(ref fields) => {
             if path.is_empty() {
-                Some(fields.keys().cloned().collect())
+                Some(
+                    fields
+                        .iter()
+                        .map(|(&ident, &id)| {
+                            // This unwrap is safe because, `id` is the field of the record
+                            // we're currently analyzing. We're sure that the linearization
+                            // phase doesn't produce wrong or invalid ids.
+                            let item = linearization.get_item(id).unwrap();
+                            let (ty, _) = linearization.resolve_item_type_meta(item);
+                            IdentWithType {
+                                ident,
+                                ty,
+                                item: Some(item.clone()),
+                            }
+                        })
+                        .collect(),
+                )
             } else {
                 let name = path.pop()?;
                 let new_id = fields.get(&name)?;
@@ -65,7 +161,7 @@ fn find_fields_from_contract(
     linearization: &Completed,
     id: ItemId,
     path: &mut Vec<Ident>,
-) -> Option<Vec<Ident>> {
+) -> Option<Vec<IdentWithType>> {
     let item = linearization.get_item(id)?;
     match &item.meta {
         Some(meta_value) => Some(find_fields_from_meta_value(meta_value, path)),
@@ -81,7 +177,10 @@ fn find_fields_from_contract(
 
 /// Find record field associated associated with a MetaValue.
 /// This can be gotten from the type or the contracts.
-fn find_fields_from_meta_value(meta_value: &MetaValue, path: &mut Vec<Ident>) -> Vec<Ident> {
+fn find_fields_from_meta_value(
+    meta_value: &MetaValue,
+    path: &mut Vec<Ident>,
+) -> Vec<IdentWithType> {
     meta_value
         .contracts
         .iter()
@@ -95,7 +194,7 @@ fn find_fields_from_meta_value(meta_value: &MetaValue, path: &mut Vec<Ident>) ->
 }
 
 /// Extract the fields from a given record type.
-fn find_fields_from_type(rrows: &RecordRows, path: &mut Vec<Ident>) -> Vec<Ident> {
+fn find_fields_from_type(rrows: &RecordRows, path: &mut Vec<Ident>) -> Vec<IdentWithType> {
     if let Some(current) = path.pop() {
         let type_of_current = rrows.iter().find_map(|item| match item {
             RecordRowsIteratorItem::Row(row) if row.id == current => Some(row.types.clone()),
@@ -115,20 +214,33 @@ fn find_fields_from_type(rrows: &RecordRows, path: &mut Vec<Ident>) -> Vec<Ident
         rrows
             .iter()
             .filter_map(|item| match item {
-                RecordRowsIteratorItem::Row(row) => Some(row.id),
+                RecordRowsIteratorItem::Row(row) => Some((row.id, row.types)),
                 _ => None,
             })
-            .collect::<Vec<_>>()
+            .map(|(ident, types)| IdentWithType {
+                ident,
+                item: None,
+                ty: types.clone(),
+            })
+            .collect()
     }
 }
 
 /// Extract record fields from a record term.
-fn find_fields_from_term(term: &RichTerm, path: &mut Vec<Ident>) -> Option<Vec<Ident>> {
+fn find_fields_from_term(term: &RichTerm, path: &mut Vec<Ident>) -> Option<Vec<IdentWithType>> {
     let current = path.pop();
     match (term.as_ref(), current) {
-        (Term::Record(data) | Term::RecRecord(data, ..), None) => {
-            Some(data.fields.keys().cloned().collect())
-        }
+        (Term::Record(data) | Term::RecRecord(data, ..), None) => Some(
+            data.fields
+                .keys()
+                .copied()
+                .map(|ident| IdentWithType {
+                    ident,
+                    ty: Types(TypeF::Flat(term.clone())),
+                    item: None,
+                })
+                .collect(),
+        ),
         (Term::Record(data) | Term::RecRecord(data, ..), Some(name)) => {
             let term = data.fields.get(&name)?;
             find_fields_from_term(term, path)
@@ -203,7 +315,11 @@ fn remove_duplicates(items: &Vec<CompletionItem>) -> Vec<CompletionItem> {
 
 /// Search the linearization to find the record information associated with a
 /// partiular ID, and in the scope of a given linearization item.
-fn collect_record_info(linearization: &Completed, id: ItemId, path: &mut Vec<Ident>) -> Vec<Ident> {
+fn collect_record_info(
+    linearization: &Completed,
+    id: ItemId,
+    path: &mut Vec<Ident>,
+) -> Vec<IdentWithType> {
     linearization
         .get_item(id)
         .map(|item| {
@@ -243,21 +359,12 @@ fn get_completion_identifiers(
         name: Ident,
         server: &Server,
         path: &mut Vec<Ident>,
-    ) -> Option<Vec<Ident>> {
+    ) -> Option<Vec<IdentWithType>> {
         let item_id = item.env.get(&name)?;
         let lin = server.lin_cache_get(&item_id.file_id).unwrap();
         Some(collect_record_info(lin, *item_id, path))
     }
 
-    /// Attach quotes to a non-ASCII string
-    fn adjust_name(name: String) -> String {
-        if name.is_ascii() {
-            name
-        } else {
-            format!("\"{}\"", name)
-        }
-    }
-
     let in_scope = match trigger {
         // Record Completion
         Some(server::DOT_COMPL_TRIGGER) => {
@@ -283,11 +390,16 @@ fn get_completion_identifiers(
                 complete(item, name, server, &mut path).unwrap_or_default()
             } else {
                 // variable name completion
+                let (ty, _) = linearization.resolve_item_type_meta(item);
                 linearization
                     .get_in_scope(item)
                     .iter()
                     .filter_map(|i| match i.kind {
-                        TermKind::Declaration(ref ident, _, _) => Some(*ident),
+                        TermKind::Declaration(ident, _, _) => Some(IdentWithType {
+                            ident,
+                            item: Some(item.clone()),
+                            ty: ty.clone(),
+                        }),
                         _ => None,
                     })
                     .collect::<Vec<_>>()
@@ -297,10 +409,7 @@ fn get_completion_identifiers(
 
     let in_scope: Vec<_> = in_scope
         .iter()
-        .map(|ident| CompletionItem {
-            label: adjust_name(ident.into_label()),
-            ..Default::default()
-        })
+        .map(|ident_meta| ident_meta.to_lsp_completion_item())
         .collect();
     Ok(remove_duplicates(&in_scope))
 }
@@ -435,12 +544,16 @@ mod tests {
         let mut path = vec![Ident::from("b"), Ident::from("a")];
         // unwrap: the conversion must succeed because we built a type without unification variable
         // nor type constants
-        let result = find_fields_from_type(&Box::new(a_record_type.try_into().unwrap()), &mut path);
-        let expected = vec![Ident::from("c1"), Ident::from("c2")];
-        assert_eq!(
-            result.iter().map(Ident::label).collect::<Vec<_>>(),
-            expected.iter().map(Ident::label).collect::<Vec<_>>()
-        )
+        let result: Vec<_> =
+            find_fields_from_type(&Box::new(a_record_type.try_into().unwrap()), &mut path)
+                .iter()
+                .map(|iwm| iwm.ident)
+                .collect();
+        let expected: Vec<_> = vec![IdentWithType::from("c1"), IdentWithType::from("c2")]
+            .iter()
+            .map(|iwm| iwm.ident)
+            .collect();
+        assert_eq!(result, expected)
     }
 
     #[test]
@@ -515,13 +628,18 @@ mod tests {
         fn single_case<const N: usize>(
             linearization: Vec<LinearizationItem<Types>>,
             ids: [ItemId; N],
-            mut expected: Vec<Ident>,
+            expected: Vec<IdentWithType>,
         ) {
+            let mut expected: Vec<_> = expected.iter().map(|iwm| iwm.ident).collect();
             expected.sort();
             let completed = make_completed(linearization);
             for id in ids {
-                let mut actual = find_fields_from_term_kind(&completed, id, &mut Vec::new())
-                    .expect("Expected Some");
+                let mut actual: Vec<_> =
+                    find_fields_from_term_kind(&completed, id, &mut Vec::new())
+                        .unwrap()
+                        .iter()
+                        .map(|iwm| iwm.ident)
+                        .collect();
                 actual.sort();
                 assert_eq!(actual, expected)
             }
@@ -560,7 +678,11 @@ mod tests {
             TermKind::Usage(UsageState::Resolved(ItemId { file_id, index: 0 })),
         );
         let linearization = vec![a, b, c, d, e];
-        let expected = vec![Ident::from("foo"), Ident::from("bar"), Ident::from("baz")];
+        let expected = vec![
+            IdentWithType::from("foo"),
+            IdentWithType::from("bar"),
+            IdentWithType::from("baz"),
+        ];
         single_case(
             linearization,
             [ItemId { file_id, index: 0 }, ItemId { file_id, index: 3 }],
@@ -622,10 +744,10 @@ mod tests {
             TermKind::Usage(UsageState::Resolved(ItemId { file_id, index: 4 })),
         );
         let expected = vec![
-            Ident::from("one"),
-            Ident::from("two"),
-            Ident::from("three"),
-            Ident::from("four"),
+            IdentWithType::from("one"),
+            IdentWithType::from("two"),
+            IdentWithType::from("three"),
+            IdentWithType::from("four"),
         ];
         let linearization = vec![a, b, c, d, e, f, g, h, i];
         single_case(
diff --git a/src/parser/grammar.lalrpop b/src/parser/grammar.lalrpop
index 7fcf877e..06310566 100644
--- a/src/parser/grammar.lalrpop
+++ b/src/parser/grammar.lalrpop
@@ -474,10 +474,15 @@ Bool: bool = {
     "false" => false,
 };
 
-// Strings that support interpolation.
+// String-like syntax which supports interpolation.
+// Depending on the opening brace, these either parse as strings, or as "symbolic strings",
+// which get desugared here to an array of terms.
 StrChunks: RichTerm = {
   <start: StringStart> <fst: ChunkLiteral?> <chunks: (ChunkExpr+ChunkLiteral)*> <lasts:ChunkExpr*> <end: StringEnd> => {
-        debug_assert_eq!(start, end);
+        debug_assert!(
+            start.is_closed_by(&end),
+            "Fatal parser error: a string starting with {start:?} should never be closed by {end:?}"
+        );
 
         let chunks: Vec<StrChunk<RichTerm>> = fst.into_iter()
             .map(StrChunk::Literal)
@@ -490,26 +495,35 @@ StrChunks: RichTerm = {
             .chain(lasts.into_iter())
             .collect();
 
-        let mut chunks = if start == StringKind::Multiline {
+        let chunks = if start.needs_strip_indent() {
             strip_indent(chunks)
-        }
-        else {
+        } else {
             chunks
         };
-        chunks.reverse();
 
-        RichTerm::from(Term::StrChunks(chunks))
+        if start == StringStartDelimiter::Symbolic {
+            let terms = chunks.into_iter().map(|chunk| match chunk {
+                StrChunk::Literal(l) => Term::Str(l).into(),
+                StrChunk::Expr(e, _) => e,
+            }).collect();
+            RichTerm::from(Term::Array(terms, Default::default()))
+        } else {
+            let mut chunks = chunks;
+            chunks.reverse();
+            RichTerm::from(Term::StrChunks(chunks))
+        }
     },
 };
 
-StringStart : StringKind = {
-    "\"" => StringKind::Standard,
-    "m%\"" => StringKind::Multiline,
+StringStart : StringStartDelimiter = {
+    "\"" => StringStartDelimiter::Standard,
+    "m%\"" => StringStartDelimiter::Multiline,
+    "s%\"" => StringStartDelimiter::Symbolic,
 };
 
-StringEnd : StringKind = {
-    "\"" => StringKind::Standard,
-    "\"%" => StringKind::Multiline,
+StringEnd : StringEndDelimiter = {
+    "\"" => StringEndDelimiter::Standard,
+    "\"%" => StringEndDelimiter::Special,
 };
 
 ChunkLiteral : String =
@@ -884,6 +898,7 @@ extern {
         "\"" => Token::Normal(NormalToken::DoubleQuote),
         "\"%" => Token::MultiStr(MultiStringToken::End),
         "m%\"" => Token::Normal(NormalToken::MultiStringStart(<usize>)),
+        "s%\"" => Token::Normal(NormalToken::SymbolicStringStart(<usize>)),
 
         "Num" => Token::Normal(NormalToken::Num),
         "Dyn" => Token::Normal(NormalToken::Dyn),
diff --git a/src/parser/lexer.rs b/src/parser/lexer.rs
index e0d021c4..32fb85e8 100644
--- a/src/parser/lexer.rs
+++ b/src/parser/lexer.rs
@@ -160,6 +160,8 @@ pub enum NormalToken<'input> {
     Underscore,
     #[regex("m(%+)\"", |lex| lex.slice().len())]
     MultiStringStart(usize),
+    #[regex("s(%+)\"", |lex| lex.slice().len())]
+    SymbolicStringStart(usize),
 
     #[token("%tag%")]
     Tag,
@@ -605,7 +607,10 @@ impl<'input> Iterator for Lexer<'input> {
             Some(Normal(NormalToken::DoubleQuote | NormalToken::StrEnumTagBegin)) => {
                 self.enter_str()
             }
-            Some(Normal(NormalToken::MultiStringStart(delim_size))) => {
+            Some(Normal(
+                NormalToken::MultiStringStart(delim_size)
+                | NormalToken::SymbolicStringStart(delim_size),
+            )) => {
                 // for interpolation & closing delimeters we only care about
                 // the number of `%`s (plus the opening `"` or `{`) so we
                 // drop the "kind marker" size here (i.e. the `m` character).
diff --git a/src/parser/utils.rs b/src/parser/utils.rs
index 46fc628f..ddbdd6fe 100644
--- a/src/parser/utils.rs
+++ b/src/parser/utils.rs
@@ -23,12 +23,39 @@ use crate::{
     types::{TypeF, Types},
 };
 
-/// Distinguish between the standard string separators `"`/`"` and the multi-line string separators
-/// `m%"`/`"%` in the parser.
+/// Distinguish between the standard string opening delimiter `"`, the multi-line string
+/// opening delimter `m%"`, and the symbolic string opening delimiter `s%"`.
 #[derive(Copy, Clone, Eq, PartialEq, Debug)]
-pub enum StringKind {
+pub enum StringStartDelimiter {
     Standard,
     Multiline,
+    Symbolic,
+}
+
+impl StringStartDelimiter {
+    pub fn is_closed_by(&self, close: &StringEndDelimiter) -> bool {
+        matches!(
+            (self, close),
+            (StringStartDelimiter::Standard, StringEndDelimiter::Standard)
+                | (StringStartDelimiter::Multiline, StringEndDelimiter::Special)
+                | (StringStartDelimiter::Symbolic, StringEndDelimiter::Special)
+        )
+    }
+
+    pub fn needs_strip_indent(&self) -> bool {
+        match self {
+            StringStartDelimiter::Standard => false,
+            StringStartDelimiter::Multiline | StringStartDelimiter::Symbolic => true,
+        }
+    }
+}
+
+/// Distinguish between the standard string closing delimter `"` and the "special" string
+/// closing delimeter `"%`.
+#[derive(Copy, Clone, Debug, Eq, PartialEq)]
+pub enum StringEndDelimiter {
+    Standard,
+    Special,
 }
 
 /// Distinguish between a normal case `id => exp` and a default case `_ => exp`.
diff --git a/src/term/mod.rs b/src/term/mod.rs
index b51d858f..92a8fb7f 100644
--- a/src/term/mod.rs
+++ b/src/term/mod.rs
@@ -390,6 +390,20 @@ impl MetaValue {
         Default::default()
     }
 
+    pub fn contracts_to_string(&self) -> Option<String> {
+        if !self.contracts.is_empty() {
+            Some(
+                self.contracts
+                    .iter()
+                    .map(|contract| format!("{}", contract.label.types,))
+                    .collect::<Vec<_>>()
+                    .join(","),
+            )
+        } else {
+            None
+        }
+    }
+
     /// Flatten two nested metavalues into one, combining their metadata. If data that can't be
     /// combined (typically, the documentation or the type annotation) are set by both metavalues,
     /// outer's one are kept.
diff --git a/src/types.rs b/src/types.rs
index 2260361e..de4f6dce 100644
--- a/src/types.rs
+++ b/src/types.rs
@@ -820,6 +820,15 @@ impl Types {
         self.subcontract(HashMap::new(), true, &mut sy)
     }
 
+    /// Returns true if this type is a function type, false otherwise.
+    pub fn is_function_type(&self) -> bool {
+        match self {
+            Types(TypeF::Forall { body, .. }) => body.is_function_type(),
+            Types(TypeF::Arrow(..)) => true,
+            Types(_) => false,
+        }
+    }
+
     /// Return the contract corresponding to a subtype.
     ///
     /// # Arguments


Test patch:
diff --git a/src/parser/tests.rs b/src/parser/tests.rs
index 4590e330..f43acbc8 100644
--- a/src/parser/tests.rs
+++ b/src/parser/tests.rs
@@ -287,69 +287,100 @@ fn invalid_record_types() {
 
 #[test]
 fn string_lexing() {
-    assert_eq!(
-        lex_without_pos("\"Good\" \"strings\""),
-        Ok(vec![
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Str(StringToken::Literal("Good")),
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Str(StringToken::Literal("strings")),
-            Token::Normal(NormalToken::DoubleQuote),
-        ])
-    );
-
-    assert_eq!(
-        lex_without_pos("\"Good\\nEscape\\t\\\"\""),
-        Ok(vec![
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Str(StringToken::Literal("Good")),
-            Token::Str(StringToken::EscapedChar('\n')),
-            Token::Str(StringToken::Literal("Escape")),
-            Token::Str(StringToken::EscapedChar('\t')),
-            Token::Str(StringToken::EscapedChar('\"')),
-            Token::Normal(NormalToken::DoubleQuote),
-        ])
-    );
-
-    assert_eq!(
-        lex_without_pos("\"1 + %{ 1 } + 2\""),
-        Ok(vec![
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Str(StringToken::Literal("1 + ")),
-            Token::Str(StringToken::Interpolation),
-            Token::Normal(NormalToken::NumLiteral(1.0)),
-            Token::Normal(NormalToken::RBrace),
-            Token::Str(StringToken::Literal(" + 2")),
-            Token::Normal(NormalToken::DoubleQuote),
-        ])
-    );
-
-    assert_eq!(
-        lex_without_pos("\"1 + %{ \"%{ 1 }\" } + 2\""),
-        Ok(vec![
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Str(StringToken::Literal("1 + ")),
-            Token::Str(StringToken::Interpolation),
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Str(StringToken::Interpolation),
-            Token::Normal(NormalToken::NumLiteral(1.0)),
-            Token::Normal(NormalToken::RBrace),
-            Token::Normal(NormalToken::DoubleQuote),
-            Token::Normal(NormalToken::RBrace),
-            Token::Str(StringToken::Literal(" + 2")),
-            Token::Normal(NormalToken::DoubleQuote),
-        ])
-    );
-
-    assert_eq!(
-        lex_without_pos(r#"m%%""%"%%"#),
-        Ok(vec![
-            Token::Normal(NormalToken::MultiStringStart(4)),
-            Token::MultiStr(MultiStringToken::Literal("\"%")),
-            Token::MultiStr(MultiStringToken::End),
-        ])
-    );
+    for (name, input, expected) in [
+        (
+            "simple strings",
+            r#""Good" "strings""#,
+            vec![
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Str(StringToken::Literal("Good")),
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Str(StringToken::Literal("strings")),
+                Token::Normal(NormalToken::DoubleQuote),
+            ],
+        ),
+        (
+            "valid escape sequence",
+            r#""Good\nEscape\t\"""#,
+            vec![
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Str(StringToken::Literal("Good")),
+                Token::Str(StringToken::EscapedChar('\n')),
+                Token::Str(StringToken::Literal("Escape")),
+                Token::Str(StringToken::EscapedChar('\t')),
+                Token::Str(StringToken::EscapedChar('\"')),
+                Token::Normal(NormalToken::DoubleQuote),
+            ],
+        ),
+        (
+            "simple interpolation",
+            r#""1 + %{ 1 } + 2""#,
+            vec![
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Str(StringToken::Literal("1 + ")),
+                Token::Str(StringToken::Interpolation),
+                Token::Normal(NormalToken::NumLiteral(1.0)),
+                Token::Normal(NormalToken::RBrace),
+                Token::Str(StringToken::Literal(" + 2")),
+                Token::Normal(NormalToken::DoubleQuote),
+            ],
+        ),
+        (
+            "nested interpolated strings",
+            r#""1 + %{ "%{ 1 }" } + 2""#,
+            vec![
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Str(StringToken::Literal("1 + ")),
+                Token::Str(StringToken::Interpolation),
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Str(StringToken::Interpolation),
+                Token::Normal(NormalToken::NumLiteral(1.0)),
+                Token::Normal(NormalToken::RBrace),
+                Token::Normal(NormalToken::DoubleQuote),
+                Token::Normal(NormalToken::RBrace),
+                Token::Str(StringToken::Literal(" + 2")),
+                Token::Normal(NormalToken::DoubleQuote),
+            ],
+        ),
+        (
+            "multiline strings only close on delmiter with correct number of %s",
+            r#"m%%""%"%%"#,
+            vec![
+                Token::Normal(NormalToken::MultiStringStart(4)),
+                Token::MultiStr(MultiStringToken::Literal("\"%")),
+                Token::MultiStr(MultiStringToken::End),
+            ],
+        ),
+        (
+            "empty symbolic string lexes like multi-line str",
+            r#"s%""%"#,
+            vec![
+                Token::Normal(NormalToken::SymbolicStringStart(3)),
+                Token::MultiStr(MultiStringToken::End),
+            ],
+        ),
+        (
+            "symbolic string with interpolation",
+            r#"s%"text %{ 1 } etc."%"#,
+            vec![
+                Token::Normal(NormalToken::SymbolicStringStart(3)),
+                Token::MultiStr(MultiStringToken::Literal("text ")),
+                Token::MultiStr(MultiStringToken::Interpolation),
+                Token::Normal(NormalToken::NumLiteral(1.0)),
+                Token::Normal(NormalToken::RBrace),
+                Token::MultiStr(MultiStringToken::Literal(" etc.")),
+                Token::MultiStr(MultiStringToken::End),
+            ],
+        ),
+    ] {
+        assert_eq!(
+            lex_without_pos(input),
+            Ok(expected),
+            "Case failed: {}",
+            name
+        )
+    }
 }
 
 #[test]
diff --git a/tests/integration/pass/symbolic-strings.ncl b/tests/integration/pass/symbolic-strings.ncl
new file mode 100644
index 00000000..9e4d7e7a
--- /dev/null
+++ b/tests/integration/pass/symbolic-strings.ncl
@@ -0,0 +1,44 @@
+let {check, ..} = import "lib/assert.ncl" in
+
+[
+  # Static symbolic string
+  s%"hello, world"% == ["hello, world"],
+  # Interpolating a string
+  let s = "test" in
+  s%"This is a %{s}"% == ["This is a ", "test"],
+  # Interpolating an interpolated string
+  let f = "f" in
+  s%"abc %{"de%{f}"}"% == ["abc ", "def"],
+  # Interpolating a number
+  s%"num: %{100}"% == ["num: ", 100],
+  # Interpolating a bool
+  s%"bool: %{true}"% == ["bool: ", true],
+  # Interpolating an array
+  s%"array: %{[true, 1, "yes"]}"% == ["array: ", [true, 1, "yes"]],
+  # Interpolating a record
+  let r = { a = 1, b = false } in
+  s%"record: %{r}"% == ["record: ", r],
+  # Interpolating multiple values
+  let str = "some string" in
+  let num = 999.999 in
+  let bool = false in
+  let array = ["an", "array", 100] in
+  let record = { a = 1, simple = "yes", record = true } in
+  let actual = s%"
+     1. %{str}
+     2. %{num}
+     3. %{bool}
+     4. %{array}
+     5. %{record}"%
+  in
+  let expected = [
+    "1. ", str,
+    "\n2. ", num,
+    "\n3. ", bool,
+    "\n4. ", array,
+    "\n5. ", record
+  ]
+  in
+  actual == expected,
+]
+|> check