Change:
<!-- Do Not Delete This! featurerequesttemplate --<!-- Please read our Rules of Conduct: --<!-- Please search existing issues to avoid creating duplicates. --<!-- Describe the feature you'd like. --Feature request for the Monaco Editor. Create an optional flag for Monaco Monarch language definitions to indicate the usage of unicode-aware regular expressions. An example of one such regular expression is , matching the '' character.
What exactly does not work? Since you can pass in regular expressions directly e.g.
Hi, What's currently not working is the unicode-aware regular expressions supported by ES2015. Even though you're able to write a RegExp including unicode-aware characters (in the format ), and even set the unicode flag for that RegExp, they don't work in Monaco Monarch at the moment. Below is the monaco-language file for ABAP, edited to support the character in identifiers. ! This is how it shows up in the editor: ! That happens precisely because the unicode flag is stripped away by compileRegExp, causing the match to fail for that character: If we change that line to, say it works as expected: ! The proposed solution is a way for the developer to tell Monaco Monarch that, instead of stripping away the unicode flag from all the regular expressions, it should instead add it to all the regular expressions if a flag is set in the language definition, just like it's the current behavior for the option. Another option I've considered was to instead use the unicode flag if, and only if, it is set in the regular expression itself, but from what I could gather, that would force us to compile the RegExp an extra time just for testing the flag, or to parse the RegExp manually. Let me know if you need more information.
:+1: Thank you, this makes good sense!

Discussion:
languageId: string; noThrow: boolean; ignoreCase: boolean; <ins> unicode: boolean; </ins> usesEmbedded: boolean; defaultToken: string; stateNames: { [stateName: string]: any; };

maxStack: number; start: string | null; ignoreCase: boolean; <ins> unicode: boolean; </ins> tokenPostfix: string; tokenizer: { [stateName: string]: IRule[]; };

// Lexer helpers /** <del> * Compiles a regular expression string, adding the 'i' flag if 'ignoreCase' is set. </del> <ins> * Compiles a regular expression string, adding the 'i' flag if 'ignoreCase' is set, and the 'u' flag if 'unicode' is set. </ins> * Also replaces @w+ or sequences with the content of the specified attribute */ function compileRegExp(lexer: monarchCommon.ILexerMin, str: string): RegExp {

}); } <del> return new RegExp(str, (lexer.ignoreCase ? 'i' : '')); </del> <ins> let flags = (lexer.ignoreCase ? 'i' : '') + (lexer.unicode ? 'u' : ''); return new RegExp(str, flags); </ins> } /**

// Set standard fields: be defensive about types lexer.start = (typeof json.start === 'string' ? json.start : null); lexer.ignoreCase = bool(json.ignoreCase, false); <ins> lexer.unicode = bool(json.unicode, false); </ins> lexer.tokenPostfix = string(json.tokenPostfix, '.' + lexer.languageId); lexer.defaultToken = string(json.defaultToken, 'source');

let lexerMin: monarchCommon.ILexerMin = <any>json; lexerMin.languageId = languageId; lexerMin.ignoreCase = lexer.ignoreCase; <ins> lexerMin.unicode = lexer.unicode; </ins> lexerMin.noThrow = lexer.noThrow; lexerMin.usesEmbedded = lexer.usesEmbedded; lexerMin.stateNames = json.tokenizer;

let regex = rule.regex; let regexSource = rule.regex.source; if (regexSource.substr(0, 4) === '^(?:' && regexSource.substr(regexSource.length - 1, 1) === ')') { <del> regex = new RegExp(regexSource.substr(4, regexSource.length - 5), regex.ignoreCase ? 'i' : ''); </del> <ins> let flags = (regex.ignoreCase ? 'i' : '') + (regex.unicode ? 'u' : ''); regex = new RegExp(regexSource.substr(4, regexSource.length - 5), flags); </ins> } let result = line.search(regex);

*/ ignoreCase?: boolean; /** <ins> * is the language unicode-aware? (i.e., /u{1D306}/) */ unicode?: boolean; /** </ins> * if no match in the tokenizer assign this token class (default 'source') */ defaultToken?: string;