Repository: jorgecarleitao/arrow2
Pull request #1024 (2022-05-30 15:20:05)

Improved performance of deserializing JSON (2x)
This PR replaces `serde_json` by `json_deserializer`, improving the performance of reading json strings by 2x and boolean by -30%, and +10% for floats:

```
read i32 2^20           time:   [68.065 ms 68.256 ms 68.444 ms]                          
                        change: [-0.8660% +1.5241% +3.8068%] (p = 0.20 > 0.05)

read f64 2^20           time:   [90.453 ms 90.682 ms 90.904 ms]                          
                        change: [+10.496% +10.879% +11.269%] (p = 0.00 < 0.05)

read utf8 2^20          time:   [51.573 ms 51.699 ms 51.825 ms]                           
                        change: [-56.883% -56.714% -56.550%] (p = 0.00 < 0.05)

read bool 2^20          time:   [33.284 ms 33.483 ms 33.837 ms]                           
                        change: [-30.126% -29.623% -28.852%] (p = 0.00 < 0.05)
```

Note that there is an API change: we now require to pass `json_deserializer::Value`, not `serde_json::Value`, to the deserializer. `json_deserializer` is exposed under `arrow2::io::json::read::json_deserializer` for convenience / version matching.


codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#1024](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (9a99a2b) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/e976c1e8874082838460314dc304369cd4b29ee9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (e976c1e) will **decrease** coverage by `0.11%`.
> The diff coverage is `50.74%`.

```diff
@@            Coverage Diff             @@
##             main    #1024      +/-   ##
==========================================
- Coverage   71.63%   71.52%   -0.12%     
==========================================
  Files         359      361       +2     
  Lines       19904    19993      +89     
==========================================
+ Hits        14259    14299      +40     
- Misses       5645     5694      +49     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/io/avro/read/decompress.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2F2cm8vcmVhZC9kZWNvbXByZXNzLnJz) | `80.00% <ø> (ø)` | |
| [src/io/avro/write/compress.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2F2cm8vd3JpdGUvY29tcHJlc3MucnM=) | `100.00% <ø> (ø)` | |
| [src/io/json/write/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2pzb24vd3JpdGUvbW9kLnJz) | `95.00% <ø> (ø)` | |
| [src/io/json\_integration/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2pzb25faW50ZWdyYXRpb24vbW9kLnJz) | `0.00% <0.00%> (ø)` | |
| [src/io/json/read/deserialize.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2pzb24vcmVhZC9kZXNlcmlhbGl6ZS5ycw==) | `61.53% <41.66%> (-6.65%)` | :arrow_down: |
| [src/io/json/write/utf8.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2pzb24vd3JpdGUvdXRmOC5ycw==) | `52.17% <52.17%> (ø)` | |
| [src/io/json/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2pzb24vbW9kLnJz) | `100.00% <100.00%> (ø)` | |
| [src/io/json/read/infer\_schema.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2pzb24vcmVhZC9pbmZlcl9zY2hlbWEucnM=) | `91.46% <100.00%> (ø)` | |
| [src/io/json/write/serialize.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2pzb24vd3JpdGUvc2VyaWFsaXplLnJz) | `87.14% <100.00%> (ø)` | |
| [src/io/ndjson/read/deserialize.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL25kanNvbi9yZWFkL2Rlc2VyaWFsaXplLnJz) | `100.00% <100.00%> (ø)` | |
| ... and [3 more](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Last update [e976c1e...9a99a2b](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1024?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/Cargo.toml b/Cargo.toml
index 09fae9293a..7b7f9c48f7 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -39,9 +39,7 @@ regex = { version = "^1.3", optional = true }
 streaming-iterator = { version = "0.1", optional = true }
 fallible-streaming-iterator = { version = "0.1", optional = true }
 
-serde = { version = "^1.0", features = ["rc"], optional = true }
-serde_derive = { version = "^1.0", optional = true }
-serde_json = { version = "^1.0", features = ["preserve_order"], optional = true }
+json-deserializer = { version = "0.3", optional = true }
 indexmap = { version = "^1.6", optional = true }
 
 # used to print columns in a nice columnar format
@@ -72,6 +70,9 @@ parquet2 = { version = "0.13", optional = true, default_features = false }
 
 # avro support
 avro-schema = { version = "0.2", optional = true }
+serde = { version = "^1.0", features = ["rc"], optional = true }
+serde_derive = { version = "^1.0", optional = true }
+serde_json = { version = "^1.0", features = ["preserve_order"], optional = true }
 # compression of avro
 libflate = { version = "1.1.1", optional = true }
 snap = { version = "1", optional = true }
@@ -134,7 +135,7 @@ io_csv_async = ["io_csv_read_async"]
 io_csv_read = ["csv", "lexical-core"]
 io_csv_read_async = ["csv-async", "lexical-core", "futures"]
 io_csv_write = ["csv-core", "streaming-iterator", "lexical-core"]
-io_json = ["serde", "serde_json", "streaming-iterator", "fallible-streaming-iterator", "indexmap", "lexical-core"]
+io_json = ["json-deserializer", "streaming-iterator", "fallible-streaming-iterator", "indexmap", "lexical-core"]
 io_ipc = ["arrow-format"]
 io_ipc_write_async = ["io_ipc", "futures"]
 io_ipc_read_async = ["io_ipc", "futures", "async-stream"]
@@ -156,9 +157,9 @@ io_avro_compression = [
     "crc",
 ]
 io_avro_async = ["io_avro", "futures", "async-stream"]
-# io_json: its dependencies + error handling
+# serde+serde_json: its dependencies + error handling
 # serde_derive: there is some derive around
-io_json_integration = ["io_json", "serde_derive", "hex"]
+io_json_integration = ["hex", "serde", "serde_derive", "serde_json", "io_ipc"]
 io_print = ["comfy-table"]
 # the compute kernels. Disabling this significantly reduces compile time.
 compute_aggregate = ["multiversion"]
@@ -313,6 +314,10 @@ harness = false
 name = "write_json"
 harness = false
 
+[[bench]]
+name = "read_json"
+harness = false
+
 [[bench]]
 name = "slices_iterator"
 harness = false
diff --git a/benches/read_json.rs b/benches/read_json.rs
new file mode 100644
index 0000000000..5d0400c592
--- /dev/null
+++ b/benches/read_json.rs
@@ -0,0 +1,67 @@
+use arrow2::array::Array;
+use arrow2::datatypes::DataType;
+use criterion::{criterion_group, criterion_main, Criterion};
+
+use arrow2::io::json::{read, write};
+use arrow2::util::bench_util::*;
+
+fn prep(array: impl Array + 'static) -> (Vec<u8>, DataType) {
+    let mut data = vec![];
+    let blocks = write::Serializer::new(
+        vec![Ok(Box::new(array) as Box<dyn Array>)].into_iter(),
+        vec![],
+    );
+    // the operation of writing is IO-bounded.
+    write::write(&mut data, blocks).unwrap();
+
+    let value = read::json_deserializer::parse(&data).unwrap();
+
+    let dt = read::infer(&value).unwrap();
+    (data, dt)
+}
+
+fn bench_read(data: &[u8], dt: &DataType) {
+    let value = read::json_deserializer::parse(data).unwrap();
+    read::deserialize(&value, dt.clone()).unwrap();
+}
+
+fn add_benchmark(c: &mut Criterion) {
+    (10..=20).step_by(2).for_each(|log2_size| {
+        let size = 2usize.pow(log2_size);
+
+        let array = create_primitive_array::<i32>(size, 0.1);
+
+        let (data, dt) = prep(array);
+
+        c.bench_function(&format!("read i32 2^{}", log2_size), |b| {
+            b.iter(|| bench_read(&data, &dt))
+        });
+
+        let array = create_primitive_array::<f64>(size, 0.1);
+
+        let (data, dt) = prep(array);
+
+        c.bench_function(&format!("read f64 2^{}", log2_size), |b| {
+            b.iter(|| bench_read(&data, &dt))
+        });
+
+        let array = create_string_array::<i32>(size, 10, 0.1, 42);
+
+        let (data, dt) = prep(array);
+
+        c.bench_function(&format!("read utf8 2^{}", log2_size), |b| {
+            b.iter(|| bench_read(&data, &dt))
+        });
+
+        let array = create_boolean_array(size, 0.1, 0.1);
+
+        let (data, dt) = prep(array);
+
+        c.bench_function(&format!("read bool 2^{}", log2_size), |b| {
+            b.iter(|| bench_read(&data, &dt))
+        });
+    })
+}
+
+criterion_group!(benches, add_benchmark);
+criterion_main!(benches);
diff --git a/examples/json_read.rs b/examples/json_read.rs
index 1630aa6504..608980a62d 100644
--- a/examples/json_read.rs
+++ b/examples/json_read.rs
@@ -1,5 +1,5 @@
-use std::fs::File;
-use std::io::BufReader;
+/// Example of reading a JSON file.
+use std::fs;
 use std::sync::Arc;
 
 use arrow2::array::Array;
@@ -7,12 +7,16 @@ use arrow2::error::Result;
 use arrow2::io::json::read;
 
 fn read_path(path: &str) -> Result<Arc<dyn Array>> {
-    // Example of reading a JSON file.
-    let reader = BufReader::new(File::open(path)?);
-    let json = serde_json::from_reader(reader)?;
+    // read the file into memory (IO-bounded)
+    let data = fs::read(path)?;
 
+    // create a non-owning struct of the data (CPU-bounded)
+    let json = read::json_deserializer::parse(&data)?;
+
+    // use it to infer an Arrow schema (CPU-bounded)
     let data_type = read::infer(&json)?;
 
+    // and deserialize it (CPU-bounded)
     read::deserialize(&json, data_type)
 }
 
diff --git a/src/io/avro/read/decompress.rs b/src/io/avro/read/decompress.rs
index 80b8153529..05f5fd43d9 100644
--- a/src/io/avro/read/decompress.rs
+++ b/src/io/avro/read/decompress.rs
@@ -9,6 +9,7 @@ use super::super::{Block, CompressedBlock};
 use super::BlockStreamIterator;
 use super::Compression;
 
+#[cfg(feature = "io_avro_compression")]
 const CRC_TABLE: crc::Crc<u32> = crc::Crc::<u32>::new(&crc::CRC_32_ISO_HDLC);
 
 /// Decompresses an Avro block.
diff --git a/src/io/avro/write/compress.rs b/src/io/avro/write/compress.rs
index 19b69460ee..5121003a2b 100644
--- a/src/io/avro/write/compress.rs
+++ b/src/io/avro/write/compress.rs
@@ -5,6 +5,7 @@ use crate::error::Result;
 use super::Compression;
 use super::{Block, CompressedBlock};
 
+#[cfg(feature = "io_avro_compression")]
 const CRC_TABLE: crc::Crc<u32> = crc::Crc::<u32>::new(&crc::CRC_32_ISO_HDLC);
 
 /// Compresses a [`Block`] to a [`CompressedBlock`].
diff --git a/src/io/json/mod.rs b/src/io/json/mod.rs
index ad406f4ffd..ebbdc92b69 100644
--- a/src/io/json/mod.rs
+++ b/src/io/json/mod.rs
@@ -5,8 +5,8 @@ pub mod write;
 
 use crate::error::Error;
 
-impl From<serde_json::error::Error> for Error {
-    fn from(error: serde_json::error::Error) -> Self {
-        Error::External("".to_string(), Box::new(error))
+impl From<json_deserializer::Error> for Error {
+    fn from(error: json_deserializer::Error) -> Self {
+        Error::ExternalFormat(error.to_string())
     }
 }
diff --git a/src/io/json/read/deserialize.rs b/src/io/json/read/deserialize.rs
index fbee18ab70..41d69f00c0 100644
--- a/src/io/json/read/deserialize.rs
+++ b/src/io/json/read/deserialize.rs
@@ -4,8 +4,7 @@ use std::{collections::hash_map::DefaultHasher, sync::Arc};
 
 use hash_hasher::HashedMap;
 use indexmap::map::IndexMap as HashMap;
-use num_traits::NumCast;
-use serde_json::Value;
+use json_deserializer::{Number, Value};
 
 use crate::{
     array::*,
@@ -17,7 +16,7 @@ use crate::{
 
 /// A function that converts a &Value into an optional tuple of a byte slice and a Value.
 /// This is used to create a dictionary, where the hashing depends on the DataType of the child object.
-type Extract = Box<dyn Fn(&Value) -> Option<(u64, &Value)>>;
+type Extract<'a> = Box<dyn Fn(&'a Value<'a>) -> Option<(u64, &'a Value<'a>)>>;
 
 fn build_extract(data_type: &DataType) -> Extract {
     match data_type {
@@ -27,11 +26,10 @@ fn build_extract(data_type: &DataType) -> Extract {
                 hasher.write(v.as_bytes());
                 Some((hasher.finish(), value))
             }
-            Value::Number(v) => v.as_f64().map(|x| {
-                let mut hasher = DefaultHasher::new();
-                hasher.write(&x.to_le_bytes());
-                (hasher.finish(), value)
-            }),
+            Value::Number(v) => match v {
+                Number::Float(_, _) => todo!(),
+                Number::Integer(_, _) => todo!(),
+            },
             Value::Bool(v) => {
                 let mut hasher = DefaultHasher::new();
                 hasher.write(&[*v as u8]);
@@ -40,25 +38,24 @@ fn build_extract(data_type: &DataType) -> Extract {
             _ => None,
         }),
         DataType::Int32 | DataType::Int64 | DataType::Int16 | DataType::Int8 => {
-            Box::new(move |value| match &value {
-                Value::Number(v) => v.as_f64().map(|x| {
+            Box::new(move |value| {
+                let integer = match value {
+                    Value::Number(number) => Some(deserialize_int_single::<i64>(*number)),
+                    Value::Bool(number) => Some(if *number { 1i64 } else { 0i64 }),
+                    _ => None,
+                };
+                integer.map(|integer| {
                     let mut hasher = DefaultHasher::new();
-                    hasher.write(&x.to_le_bytes());
+                    hasher.write(&integer.to_le_bytes());
                     (hasher.finish(), value)
-                }),
-                Value::Bool(v) => {
-                    let mut hasher = DefaultHasher::new();
-                    hasher.write(&[*v as u8]);
-                    Some((hasher.finish(), value))
-                }
-                _ => None,
+                })
             })
         }
         _ => Box::new(|_| None),
     }
 }
 
-fn deserialize_boolean<A: Borrow<Value>>(rows: &[A]) -> BooleanArray {
+fn deserialize_boolean<'a, A: Borrow<Value<'a>>>(rows: &[A]) -> BooleanArray {
     let iter = rows.iter().map(|row| match row.borrow() {
         Value::Bool(v) => Some(v),
         _ => None,
@@ -66,31 +63,125 @@ fn deserialize_boolean<A: Borrow<Value>>(rows: &[A]) -> BooleanArray {
     BooleanArray::from_trusted_len_iter(iter)
 }
 
-fn deserialize_int<T: NativeType + NumCast, A: Borrow<Value>>(
+fn deserialize_int_single<T>(number: Number) -> T
+where
+    T: NativeType + lexical_core::FromLexical + Pow10,
+{
+    match number {
+        Number::Float(fraction, exponent) => {
+            let integer = fraction.split(|x| *x == b'.').next().unwrap();
+            let mut integer: T = lexical_core::parse(integer).unwrap();
+            if !exponent.is_empty() {
+                let exponent: u32 = lexical_core::parse(exponent).unwrap();
+                integer = integer.pow10(exponent);
+            }
+            integer
+        }
+        Number::Integer(integer, exponent) => {
+            let mut integer: T = lexical_core::parse(integer).unwrap();
+            if !exponent.is_empty() {
+                let exponent: u32 = lexical_core::parse(exponent).unwrap();
+                integer = integer.pow10(exponent);
+            }
+            integer
+        }
+    }
+}
+
+trait Powi10: NativeType + num_traits::One + std::ops::Add {
+    fn powi10(self, exp: i32) -> Self;
+}
+
+impl Powi10 for f32 {
+    #[inline]
+    fn powi10(self, exp: i32) -> Self {
+        self * 10.0f32.powi(exp)
+    }
+}
+
+impl Powi10 for f64 {
+    #[inline]
+    fn powi10(self, exp: i32) -> Self {
+        self * 10.0f64.powi(exp)
+    }
+}
+
+trait Pow10: NativeType + num_traits::One + std::ops::Add {
+    fn pow10(self, exp: u32) -> Self;
+}
+
+macro_rules! impl_pow10 {
+    ($ty:ty) => {
+        impl Pow10 for $ty {
+            #[inline]
+            fn pow10(self, exp: u32) -> Self {
+                self * (10 as $ty).pow(exp)
+            }
+        }
+    };
+}
+impl_pow10!(u8);
+impl_pow10!(u16);
+impl_pow10!(u32);
+impl_pow10!(u64);
+impl_pow10!(i8);
+impl_pow10!(i16);
+impl_pow10!(i32);
+impl_pow10!(i64);
+
+fn deserialize_float_single<T>(number: &Number) -> T
+where
+    T: NativeType + lexical_core::FromLexical + Powi10,
+{
+    match number {
+        Number::Float(float, exponent) => {
+            let mut float: T = lexical_core::parse(float).unwrap();
+            if !exponent.is_empty() {
+                let exponent: i32 = lexical_core::parse(exponent).unwrap();
+                float = float.powi10(exponent);
+            }
+            float
+        }
+        Number::Integer(integer, exponent) => {
+            let mut float: T = lexical_core::parse(integer).unwrap();
+            if !exponent.is_empty() {
+                let exponent: i32 = lexical_core::parse(exponent).unwrap();
+                float = float.powi10(exponent);
+            }
+            float
+        }
+    }
+}
+
+fn deserialize_int<'a, T: NativeType + lexical_core::FromLexical + Pow10, A: Borrow<Value<'a>>>(
     rows: &[A],
     data_type: DataType,
 ) -> PrimitiveArray<T> {
     let iter = rows.iter().map(|row| match row.borrow() {
-        Value::Number(number) => number.as_i64().and_then(num_traits::cast::<i64, T>),
-        Value::Bool(number) => num_traits::cast::<i32, T>(*number as i32),
+        Value::Number(number) => Some(deserialize_int_single(*number)),
+        Value::Bool(number) => Some(if *number { T::one() } else { T::default() }),
         _ => None,
     });
     PrimitiveArray::from_trusted_len_iter(iter).to(data_type)
 }
 
-fn deserialize_float<T: NativeType + NumCast, A: Borrow<Value>>(
+fn deserialize_float<
+    'a,
+    T: NativeType + lexical_core::FromLexical + Powi10,
+    A: Borrow<Value<'a>>,
+>(
     rows: &[A],
     data_type: DataType,
 ) -> PrimitiveArray<T> {
     let iter = rows.iter().map(|row| match row.borrow() {
-        Value::Number(number) => number.as_f64().and_then(num_traits::cast::<f64, T>),
-        Value::Bool(number) => num_traits::cast::<i32, T>(*number as i32),
+        Value::Number(number) => Some(deserialize_float_single(number)),
+        Value::Bool(number) => Some(if *number { T::one() } else { T::default() }),
         _ => None,
     });
     PrimitiveArray::from_trusted_len_iter(iter).to(data_type)
 }
 
-fn deserialize_binary<O: Offset, A: Borrow<Value>>(rows: &[A]) -> BinaryArray<O> {
+fn deserialize_binary<'a, O: Offset, A: Borrow<Value<'a>>>(rows: &[A]) -> BinaryArray<O> {
     let iter = rows.iter().map(|row| match row.borrow() {
         Value::String(v) => Some(v.as_bytes()),
         _ => None,
@@ -98,17 +189,31 @@ fn deserialize_binary<O: Offset, A: Borrow<Value>>(rows: &[A]) -> BinaryArray<O>
     BinaryArray::from_trusted_len_iter(iter)
 }
 
-fn deserialize_utf8<O: Offset, A: Borrow<Value>>(rows: &[A]) -> Utf8Array<O> {
-    let iter = rows.iter().map(|row| match row.borrow() {
-        Value::String(v) => Some(v.clone()),
-        Value::Number(v) => Some(v.to_string()),
-        Value::Bool(v) => Some(v.to_string()),
-        _ => None,
-    });
-    Utf8Array::<O>::from_trusted_len_iter(iter)
+fn deserialize_utf8<'a, O: Offset, A: Borrow<Value<'a>>>(rows: &[A]) -> Utf8Array<O> {
+    let mut array = MutableUtf8Array::<O>::with_capacity(rows.len());
+    let mut scratch = vec![];
+    for row in rows {
+        match row.borrow() {
+            Value::String(v) => array.push(Some(v.as_ref())),
+            Value::Number(number) => match number {
+                Number::Integer(number, exponent) | Number::Float(number, exponent) => {
+                    scratch.clear();
+                    scratch.extend_from_slice(*number);
+                    scratch.push(b'e');
+                    scratch.extend_from_slice(*exponent);
+                }
+            },
+            Value::Bool(v) => array.push(Some(if *v { "true" } else { "false" })),
+            _ => array.push_null(),
+        }
+    }
+    array.into()
 }
 
-fn deserialize_list<O: Offset, A: Borrow<Value>>(rows: &[A], data_type: DataType) -> ListArray<O> {
+fn deserialize_list<'a, O: Offset, A: Borrow<Value<'a>>>(
+    rows: &[A],
+    data_type: DataType,
+) -> ListArray<O> {
     let child = ListArray::<O>::get_child_type(&data_type);
 
     let mut validity = MutableBitmap::with_capacity(rows.len());
@@ -138,7 +243,7 @@ fn deserialize_list<O: Offset, A: Borrow<Value>>(rows: &[A], data_type: DataType
     ListArray::<O>::new(data_type, offsets.into(), values, validity.into())
 }
 
-fn deserialize_struct<A: Borrow<Value>>(rows: &[A], data_type: DataType) -> StructArray {
+fn deserialize_struct<'a, A: Borrow<Value<'a>>>(rows: &[A], data_type: DataType) -> StructArray {
     let fields = StructArray::get_fields(&data_type);
 
     let mut values = fields
@@ -173,7 +278,7 @@ fn deserialize_struct<A: Borrow<Value>>(rows: &[A], data_type: DataType) -> Stru
     StructArray::new(data_type, values, validity.into())
 }
 
-fn deserialize_dictionary<K: DictionaryKey, A: Borrow<Value>>(
+fn deserialize_dictionary<'a, K: DictionaryKey, A: Borrow<Value<'a>>>(
     rows: &[A],
     data_type: DataType,
 ) -> DictionaryArray<K> {
@@ -206,7 +311,10 @@ fn deserialize_dictionary<K: DictionaryKey, A: Borrow<Value>>(
     DictionaryArray::<K>::from_data(keys, values)
 }
 
-pub(crate) fn _deserialize<A: Borrow<Value>>(rows: &[A], data_type: DataType) -> Arc<dyn Array> {
+pub(crate) fn _deserialize<'a, A: Borrow<Value<'a>>>(
+    rows: &[A],
+    data_type: DataType,
+) -> Arc<dyn Array> {
     match &data_type {
         DataType::Null => Arc::new(NullArray::new(data_type, rows.len())),
         DataType::Boolean => Arc::new(deserialize_boolean(rows)),
diff --git a/src/io/json/read/infer_schema.rs b/src/io/json/read/infer_schema.rs
index 336264c431..2ea4f84058 100644
--- a/src/io/json/read/infer_schema.rs
+++ b/src/io/json/read/infer_schema.rs
@@ -1,8 +1,9 @@
 use std::borrow::Borrow;
+use std::collections::BTreeMap;
 
 use indexmap::map::IndexMap as HashMap;
 use indexmap::set::IndexSet as HashSet;
-use serde_json::Value;
+use json_deserializer::{Number, Value};
 
 use crate::datatypes::*;
 use crate::error::Result;
@@ -29,7 +30,7 @@ fn filter_map_nulls(dt: DataType) -> Option<DataType> {
     }
 }
 
-fn infer_object(inner: &serde_json::Map<String, Value>) -> Result<DataType> {
+fn infer_object(inner: &BTreeMap<String, Value>) -> Result<DataType> {
     let fields = inner
         .iter()
         .filter_map(|(key, value)| {
@@ -69,11 +70,10 @@ fn infer_array(values: &[Value]) -> Result<DataType> {
     })
 }
 
-fn infer_number(n: &serde_json::Number) -> DataType {
-    if n.is_f64() {
-        DataType::Float64
-    } else {
-        DataType::Int64
+fn infer_number(n: &Number) -> DataType {
+    match n {
+        Number::Float(..) => DataType::Float64,
+        Number::Integer(..) => DataType::Int64,
     }
 }
 
diff --git a/src/io/json/read/mod.rs b/src/io/json/read/mod.rs
index 908da67b51..ed1ad17f10 100644
--- a/src/io/json/read/mod.rs
+++ b/src/io/json/read/mod.rs
@@ -6,3 +6,5 @@ pub(crate) use deserialize::_deserialize;
 pub use deserialize::deserialize;
 pub(crate) use infer_schema::coerce_data_type;
 pub use infer_schema::infer;
+
+pub use json_deserializer;
diff --git a/src/io/json/write/mod.rs b/src/io/json/write/mod.rs
index 8a416819f7..2278a636f2 100644
--- a/src/io/json/write/mod.rs
+++ b/src/io/json/write/mod.rs
@@ -1,5 +1,6 @@
 //! APIs to write to JSON
 mod serialize;
+mod utf8;
 
 pub use fallible_streaming_iterator::*;
 pub(crate) use serialize::new_serializer;
diff --git a/src/io/json/write/serialize.rs b/src/io/json/write/serialize.rs
index fb90c17ad0..afaf7dab04 100644
--- a/src/io/json/write/serialize.rs
+++ b/src/io/json/write/serialize.rs
@@ -13,6 +13,8 @@ use crate::temporal_conversions::{
 use crate::util::lexical_to_bytes_mut;
 use crate::{array::*, datatypes::DataType, types::NativeType};
 
+use super::utf8;
+
 fn boolean_serializer<'a>(
     array: &'a BooleanArray,
 ) -> Box<dyn StreamingIterator<Item = [u8]> + 'a + Send + Sync> {
@@ -73,7 +75,7 @@ fn utf8_serializer<'a, O: Offset>(
         array.iter(),
         |x, buf| {
             if let Some(x) = x {
-                serde_json::to_writer(buf, x).unwrap();
+                utf8::write_str(buf, x).unwrap();
             } else {
                 buf.extend_from_slice(b"null")
             }
@@ -256,7 +258,7 @@ fn serialize_item(buffer: &mut Vec<u8>, record: &[(&str, &[u8])], is_first_row:
             buffer.push(b',');
         }
         first_item = false;
-        serde_json::to_writer(&mut *buffer, key).unwrap();
+        utf8::write_str(buffer, key).unwrap();
         buffer.push(b':');
         buffer.extend(*value);
     }
diff --git a/src/io/json/write/utf8.rs b/src/io/json/write/utf8.rs
new file mode 100644
index 0000000000..b8c9852217
--- /dev/null
+++ b/src/io/json/write/utf8.rs
@@ -0,0 +1,138 @@
+// Adapted from https://github.com/serde-rs/json/blob/f901012df66811354cb1d490ad59480d8fdf77b5/src/ser.rs
+use std::io;
+
+pub fn write_str<W>(writer: &mut W, value: &str) -> io::Result<()>
+where
+    W: io::Write,
+{
+    writer.write_all(b"\"")?;
+    let bytes = value.as_bytes();
+
+    let mut start = 0;
+
+    for (i, &byte) in bytes.iter().enumerate() {
+        let escape = ESCAPE[byte as usize];
+        if escape == 0 {
+            continue;
+        }
+
+        if start < i {
+            writer.write_all(&bytes[start..i])?;
+        }
+
+        let char_escape = CharEscape::from_escape_table(escape, byte);
+        write_char_escape(writer, char_escape)?;
+
+        start = i + 1;
+    }
+
+    if start != bytes.len() {
+        writer.write_all(&bytes[start..])?;
+    }
+    writer.write_all(b"\"")
+}
+
+const BB: u8 = b'b'; // \x08
+const TT: u8 = b't'; // \x09
+const NN: u8 = b'n'; // \x0A
+const FF: u8 = b'f'; // \x0C
+const RR: u8 = b'r'; // \x0D
+const QU: u8 = b'"'; // \x22
+const BS: u8 = b'\\'; // \x5C
+const UU: u8 = b'u'; // \x00...\x1F except the ones above
+const __: u8 = 0;
+
+// Lookup table of escape sequences. A value of b'x' at index i means that byte
+// i is escaped as "\x" in JSON. A value of 0 means that byte i is not escaped.
+static ESCAPE: [u8; 256] = [
+    //   1   2   3   4   5   6   7   8   9   A   B   C   D   E   F
+    UU, UU, UU, UU, UU, UU, UU, UU, BB, TT, NN, UU, FF, RR, UU, UU, // 0
+    UU, UU, UU, UU, UU, UU, UU, UU, UU, UU, UU, UU, UU, UU, UU, UU, // 1
+    __, __, QU, __, __, __, __, __, __, __, __, __, __, __, __, __, // 2
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 3
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 4
+    __, __, __, __, __, __, __, __, __, __, __, __, BS, __, __, __, // 5
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 6
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 7
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 8
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // 9
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // A
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // B
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // C
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // D
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // E
+    __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, __, // F
+];
+
+/// Represents a character escape code in a type-safe manner.
+pub enum CharEscape {
+    /// An escaped quote `"`
+    Quote,
+    /// An escaped reverse solidus `\`
+    ReverseSolidus,
+    // An escaped solidus `/`
+    //Solidus,
+    /// An escaped backspace character (usually escaped as `\b`)
+    Backspace,
+    /// An escaped form feed character (usually escaped as `\f`)
+    FormFeed,
+    /// An escaped line feed character (usually escaped as `\n`)
+    LineFeed,
+    /// An escaped carriage return character (usually escaped as `\r`)
+    CarriageReturn,
+    /// An escaped tab character (usually escaped as `\t`)
+    Tab,
+    /// An escaped ASCII plane control character (usually escaped as
+    /// `\u00XX` where `XX` are two hex characters)
+    AsciiControl(u8),
+}
+
+impl CharEscape {
+    #[inline]
+    fn from_escape_table(escape: u8, byte: u8) -> CharEscape {
+        match escape {
+            self::BB => CharEscape::Backspace,
+            self::TT => CharEscape::Tab,
+            self::NN => CharEscape::LineFeed,
+            self::FF => CharEscape::FormFeed,
+            self::RR => CharEscape::CarriageReturn,
+            self::QU => CharEscape::Quote,
+            self::BS => CharEscape::ReverseSolidus,
+            self::UU => CharEscape::AsciiControl(byte),
+            _ => unreachable!(),
+        }
+    }
+}
+
+#[inline]
+fn write_char_escape<W>(writer: &mut W, char_escape: CharEscape) -> io::Result<()>
+where
+    W: io::Write,
+{
+    use self::CharEscape::*;
+
+    let s = match char_escape {
+        Quote => b"\\\"",
+        ReverseSolidus => b"\\\\",
+        //Solidus => b"\\/",
+        Backspace => b"\\b",
+        FormFeed => b"\\f",
+        LineFeed => b"\\n",
+        CarriageReturn => b"\\r",
+        Tab => b"\\t",
+        AsciiControl(byte) => {
+            static HEX_DIGITS: [u8; 16] = *b"0123456789abcdef";
+            let bytes = &[
+                b'\\',
+                b'u',
+                b'0',
+                b'0',
+                HEX_DIGITS[(byte >> 4) as usize],
+                HEX_DIGITS[(byte & 0xF) as usize],
+            ];
+            return writer.write_all(bytes);
+        }
+    };
+
+    writer.write_all(s)
+}
diff --git a/src/io/json_integration/mod.rs b/src/io/json_integration/mod.rs
index 279f2ec55f..dc522c60c8 100644
--- a/src/io/json_integration/mod.rs
+++ b/src/io/json_integration/mod.rs
@@ -5,6 +5,8 @@
 use serde_derive::{Deserialize, Serialize};
 use serde_json::Value;
 
+use crate::error::Error;
+
 pub mod read;
 pub mod write;
 
@@ -117,3 +119,9 @@ pub struct ArrowJsonColumn {
     /// the children
     pub children: Option<Vec<ArrowJsonColumn>>,
 }
+
+impl From<serde_json::Error> for Error {
+    fn from(error: serde_json::Error) -> Self {
+        Error::ExternalFormat(error.to_string())
+    }
+}
diff --git a/src/io/ndjson/read/deserialize.rs b/src/io/ndjson/read/deserialize.rs
index 49e1ef6492..dae87d66d7 100644
--- a/src/io/ndjson/read/deserialize.rs
+++ b/src/io/ndjson/read/deserialize.rs
@@ -1,6 +1,6 @@
 use std::sync::Arc;
 
-use serde_json::Value;
+use json_deserializer::parse;
 
 use crate::array::Array;
 use crate::datatypes::DataType;
@@ -22,7 +22,7 @@ pub fn deserialize(rows: &[String], data_type: DataType) -> Result<Arc<dyn Array
         ));
     }
 
-    deserialize_iter(rows.iter(), data_type)
+    deserialize_iter(rows.iter().map(|x| x.as_ref()), data_type)
 }
 
 /// Deserializes an iterator of rows into an [`Array`] of [`DataType`].
@@ -31,14 +31,14 @@ pub fn deserialize(rows: &[String], data_type: DataType) -> Result<Arc<dyn Array
 /// This function is guaranteed to return an array of length equal to the leng
 /// # Errors
 /// This function errors iff any of the rows is not a valid JSON (i.e. the format is not valid NDJSON).
-pub fn deserialize_iter<A: AsRef<str>>(
-    rows: impl Iterator<Item = A>,
+pub fn deserialize_iter<'a>(
+    rows: impl Iterator<Item = &'a str>,
     data_type: DataType,
 ) -> Result<Arc<dyn Array>, Error> {
     // deserialize strings to `Value`s
     let rows = rows
-        .map(|row| serde_json::from_str(row.as_ref()).map_err(Error::from))
-        .collect::<Result<Vec<Value>, Error>>()?;
+        .map(|row| parse(row.as_bytes()).map_err(Error::from))
+        .collect::<Result<Vec<_>, Error>>()?;
 
     // deserialize &[Value] to Array
     Ok(_deserialize(&rows, data_type))
diff --git a/src/io/ndjson/read/file.rs b/src/io/ndjson/read/file.rs
index 2b0174a02b..6ed5497489 100644
--- a/src/io/ndjson/read/file.rs
+++ b/src/io/ndjson/read/file.rs
@@ -2,8 +2,7 @@ use std::io::BufRead;
 
 use fallible_streaming_iterator::FallibleStreamingIterator;
 use indexmap::set::IndexSet as HashSet;
-use serde_json;
-use serde_json::Value;
+use json_deserializer::parse;
 
 use crate::{
     datatypes::DataType,
@@ -115,7 +114,7 @@ pub fn infer<R: std::io::BufRead>(
 
     let mut data_types = HashSet::new();
     while let Some(rows) = reader.next()? {
-        let value: Value = serde_json::from_str(&rows[0])?; // 0 because it is row by row
+        let value = parse(rows[0].as_bytes())?; // 0 because it is row by row
         let data_type = infer_json(&value)?;
         if data_type != DataType::Null {
             data_types.insert(data_type);
@@ -134,7 +133,7 @@ pub fn infer<R: std::io::BufRead>(
 pub fn infer_iter<A: AsRef<str>>(rows: impl Iterator<Item = A>) -> Result<DataType> {
     let mut data_types = HashSet::new();
     for row in rows {
-        let v: Value = serde_json::from_str(row.as_ref())?;
+        let v = parse(row.as_ref().as_bytes())?;
         let data_type = infer_json(&v)?;
         if data_type != DataType::Null {
             data_types.insert(data_type);


Test patch:
diff --git a/tests/it/io/json/read.rs b/tests/it/io/json/read.rs
index db1b067bb4..e584324931 100644
--- a/tests/it/io/json/read.rs
+++ b/tests/it/io/json/read.rs
@@ -7,7 +7,7 @@ use super::*;
 
 #[test]
 fn read_json() -> Result<()> {
-    let data = r#"[
+    let data = br#"[
         {
             "a": 1
         },
@@ -19,7 +19,7 @@ fn read_json() -> Result<()> {
         }
     ]"#;
 
-    let json = serde_json::from_slice(data.as_bytes())?;
+    let json = json_deserializer::parse(data)?;
 
     let data_type = read::infer(&json)?;
 
diff --git a/tests/it/io/ndjson/read.rs b/tests/it/io/ndjson/read.rs
index 66ea5e0aba..ef3e6882e5 100644
--- a/tests/it/io/ndjson/read.rs
+++ b/tests/it/io/ndjson/read.rs
@@ -156,7 +156,7 @@ fn invalid_infer_schema() -> Result<()> {
     let re = ndjson_read::infer(&mut Cursor::new("city,lat,lng"), None);
     assert_eq!(
         re.err().unwrap().to_string(),
-        "External error: expected value at line 1 column 1",
+        "External format error: InvalidToken(99)",
     );
     Ok(())
 }
@@ -249,7 +249,7 @@ fn invalid_read_record() -> Result<()> {
 
     assert_eq!(
         arrays.err().unwrap().to_string(),
-        "External error: expected value at line 1 column 1",
+        "External format error: InvalidToken(99)",
     );
     Ok(())
 }