Repository: joernio/joern (scala)

Issue:
[Bug] Erroneous AST parsing in JSSRC
Parsing a (possibly ill-formatted) TypeScript file results in a java.lang.NumberFormatException.

**To Reproduce**
Steps to reproduce the behavior:
1. import the attached example with `importCode` in joern
[example.zip](https://github.com/joernio/joern/files/11112423/example.zip)

2. Observe the following errors:
```java
2023-03-30 13:49:34.946 WARN AstGenRunner$: 	- failed to parse '/home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/retroactiveVesting.ts': 'Unexpected token, expected "}" (39:6)'
java.lang.NumberFormatException: 100000000000000000000
	at upickle.core.Util$.parseLong(Util.scala:83)
	at upickle.core.Util$.parseIntegralNum(Util.scala:48)
	at ujson.Value$.visitFloat64StringParts(Value.scala:213)
```

<details><summary>Full Stack Trace</summary>
<p>

```java
Using generator for language: JSSRC: JsSrcCpgGenerator
Creating project `0xlenin_saddle-contract2` for code at `/home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/`
=======================================================================================================
Invoking CPG generator in a separate process. Note that the new process will consume additional memory.
If you are importing a large codebase (and/or running into memory issues), please try the following:
1) exit joern
2) invoke the frontend: /opt/joern/joern-cli/jssrc2cpg.sh -J-Xmx30688m /home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/ --output /home/pr0me/workspace/0xlenin_saddle-contract2/cpg.bin.zip
3) start joern, import the cpg: `importCpg("path/to/cpg")`
=======================================================================================================

2023-03-30 13:49:34.946 WARN AstGenRunner$: 	- failed to parse '/home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/retroactiveVesting.ts': 'Unexpected token, expected "}" (39:6)'
java.lang.NumberFormatException: 100000000000000000000
	at upickle.core.Util$.parseLong(Util.scala:83)
	at upickle.core.Util$.parseIntegralNum(Util.scala:48)
	at ujson.Value$.visitFloat64StringParts(Value.scala:213)
	at ujson.Value$.visitFloat64StringParts(Value.scala:129)
	at ujson.CharParser.visitFloat64StringPartsWithWrapper(CharParser.scala:161)
	at ujson.CharParser.parseNum(CharParser.scala:148)
	at ujson.CharParser.parseNested(CharParser.scala:409)
	at ujson.CharParser.parseTopLevel0(CharParser.scala:324)
	at ujson.CharParser.parseTopLevel(CharParser.scala:308)
	at ujson.CharParser.parse(CharParser.scala:59)
	at ujson.StringParser$.transform(StringParser.scala:28)
	at ujson.StringParser$.transform(StringParser.scala:27)
	at ujson.Readable$fromTransformer.transform(Readable.scala:13)
	at ujson.package$.transform(package.scala:4)
	at ujson.package$.$anonfun$read$1(package.scala:10)
	at upickle.core.TraceVisitor$.withTrace(TraceVisitor.scala:18)
	at ujson.package$.read(package.scala:10)
	at io.joern.jssrc2cpg.parser.BabelJsonParser$.readFile(BabelJsonParser.scala:31)
	at io.joern.jssrc2cpg.passes.AstCreationPass.$anonfun$runOnPart$1(AstCreationPass.scala:46)
	at io.joern.jssrc2cpg.utils.TimeUtils$.time(TimeUtils.scala:11)
	at io.joern.jssrc2cpg.passes.AstCreationPass.runOnPart(AstCreationPass.scala:45)
	at io.joern.jssrc2cpg.passes.AstCreationPass.runOnPart(AstCreationPass.scala:22)
	at io.shiftleft.passes.ConcurrentWriterCpgPass.$anonfun$createApplySerializeAndStore$1(ParallelCpgPass.scala:95)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
moving cpg.bin.zip to cpg.bin because it is already a database file
Creating working copy of CPG to be safe
Loading base CPG from: /home/pr0me/workspace/0xlenin_saddle-contract2/cpg.bin.tmp
Code successfully imported. You can now query it using `cpg`.
For an overview of all imported code, type `workspace`.
Missing metaData block
Adding default overlays to base CPG
Missing metaData block
Missing metaData block
Missing metaData block
Missing metaData block
Missing metaData block
2023-03-30 13:49:36.204 WARN ControlFlow: io.joern.x2cpg.layers.ControlFlow depends on List(base) but CPG only has Set() - skipping creation
Missing metaData block
Missing metaData block
2023-03-30 13:49:36.207 WARN TypeRelations: io.joern.x2cpg.layers.TypeRelations depends on List(base) but CPG only has Set() - skipping creation
Missing metaData block
Missing metaData block
2023-03-30 13:49:36.207 WARN CallGraph: io.joern.x2cpg.layers.CallGraph depends on List(typerel) but CPG only has Set() - skipping creation
The graph has been modified. You may want to use the `save` command to persist changes to disk.  All changes will also be saved collectively on exit
Missing metaData block
Missing metaData block
2023-03-30 13:49:36,359 main ERROR An exception occurred processing Appender Console org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.NoSuchMethodError: 'java.util.Deque org.apache.logging.log4j.util.StackLocatorUtil.getCurrentStackTrace()'

[...]
```

</p>
</details> 

3. From the stack trace:
Note that there are multiple missing blocks and structures, probably due to the failure in parsing the AST.
(The error at the bottom is discussed in issue #2439 )

**Expected behavior**
AST generation should succeed, fail gracefully if possible or abort CPG creation completely if unrecoverable.

**Desktop (please complete the following information):**
- OS: Linux 5.15.0-1030-gcp 37-Ubuntu
- Joern Version: 1.1.1552
- javac 11.0.18

Pull request:
We handle the NumberFormatException in a custom visitor for `ujson.transform` now.

Fixes: https://github.com/joernio/joern/issues/2440

Patch:
diff --git a/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/parser/BabelJsonParser.scala b/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/parser/BabelJsonParser.scala
index 9beb5a55f540..14ea2a9a94e6 100644
--- a/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/parser/BabelJsonParser.scala
+++ b/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/parser/BabelJsonParser.scala
@@ -28,7 +28,7 @@ object BabelJsonParser {
     }
 
     val jsonContent       = IOUtils.readLinesInFile(file).mkString
-    val json              = ujson.read(jsonContent)
+    val json              = ujson.transform(jsonContent, JsValueVisitor)
     val filename          = json("relativeName").str
     val fullPath          = Paths.get(rootPath.toString, filename)
     val ls                = SourceFiles.retrieveLineSeparator(fullPath.toString)
diff --git a/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/parser/JsValueVisitor.scala b/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/parser/JsValueVisitor.scala
new file mode 100644
index 000000000000..f4f0c53994d0
--- /dev/null
+++ b/joern-cli/frontends/jssrc2cpg/src/main/scala/io/joern/jssrc2cpg/parser/JsValueVisitor.scala
@@ -0,0 +1,70 @@
+package io.joern.jssrc2cpg.parser
+
+import ujson.AstTransformer
+import ujson.Value
+import upickle.core.Util
+import upickle.core.Visitor
+import upickle.core.compat.Factory
+import upickle.core.ArrVisitor
+import upickle.core.ObjVisitor
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+import scala.util.Failure
+import scala.util.Success
+import scala.util.Try
+
+/** Custom visitor that mimics the behavior of ujson.Value. Provides a fix for:
+  * https://github.com/joernio/joern/issues/2440
+  */
+object JsValueVisitor extends AstTransformer[Value] {
+
+  private val adapted = Value
+
+  private class JsAstObjVisitor[T](build: T => Value)(implicit factory: Factory[(String, Value), T])
+      extends adapted.AstObjVisitor(build) {
+    override def subVisitor: JsValueVisitor.type = JsValueVisitor.this
+  }
+
+  private class JsAstArrVisitor[T[_]](build: T[Value] => Value)(implicit factory: Factory[Value, T[Value]])
+      extends AstArrVisitor(build) {
+    override def subVisitor: JsValueVisitor.type = JsValueVisitor.this
+  }
+
+  override def transform[T](j: Value, f: Visitor[_, T]): T =
+    adapted.transform(j, f)
+
+  override def visitJsonableObject(length: Int, index: Int): ObjVisitor[Value, Value] =
+    new JsAstObjVisitor[mutable.LinkedHashMap[String, Value]](xs => ujson.Obj(xs))
+
+  override def visitArray(length: Int, index: Int): ArrVisitor[Value, Value] =
+    new JsAstArrVisitor[ArrayBuffer](xs => ujson.Arr(xs))
+
+  override def visitNull(index: Int): Value =
+    adapted.visitNull(index)
+
+  override def visitFalse(index: Int): Value =
+    adapted.visitFalse(index)
+
+  override def visitTrue(index: Int): Value =
+    adapted.visitTrue(index)
+
+  override def visitString(s: CharSequence, index: Int): Value =
+    adapted.visitString(s, index)
+
+  override def visitFloat64StringParts(s: CharSequence, decIndex: Int, expIndex: Int, index: Int): Value = {
+    val value: Double = if (decIndex != -1 || expIndex != -1) {
+      s.toString.toDouble
+    } else {
+      // We accept the loss of precision here.
+      // See: https://github.com/joernio/joern/issues/2440
+      Try(Util.parseIntegralNum(s, decIndex, expIndex, index)) match {
+        case Success(num)                      => num.toDouble
+        case Failure(_: NumberFormatException) => s.toString.toDouble
+        case Failure(other: Throwable)         => throw other
+      }
+    }
+    ujson.Num(value)
+  }
+
+}


Test patch:
diff --git a/joern-cli/frontends/jssrc2cpg/src/test/scala/io/joern/jssrc2cpg/passes/ast/SimpleAstCreationPassTest.scala b/joern-cli/frontends/jssrc2cpg/src/test/scala/io/joern/jssrc2cpg/passes/ast/SimpleAstCreationPassTest.scala
index e1c6115ca5b1..3640ac713e13 100644
--- a/joern-cli/frontends/jssrc2cpg/src/test/scala/io/joern/jssrc2cpg/passes/ast/SimpleAstCreationPassTest.scala
+++ b/joern-cli/frontends/jssrc2cpg/src/test/scala/io/joern/jssrc2cpg/passes/ast/SimpleAstCreationPassTest.scala
@@ -9,6 +9,11 @@ class SimpleAstCreationPassTest extends AbstractPassTest {
 
   "AST generation for simple fragments" should {
 
+    "have correct structure for long numeric literal" in AstFixture("console.log(1e20)") { cpg =>
+      val List(lit) = cpg.literal.l
+      lit.code shouldBe "1e20"
+    }
+
     "have correct structure for non null expression" in AstFixture("const foo = bar!") { cpg =>
       val List(nonNullCall) = cpg.call(Operators.notNullAssert).l
       val List(arg)         = nonNullCall.argument.isIdentifier.l