fabiochiu/medium-articles huggingface dataset:
Calling JSON API and manipulating result with Node.js
By: Fabio Brigati
Tags: API, Nodejs, JavaScript, Json

Write the Javascript code

Within your project’s folder create a file (ie.: ‘fetchData.js’), and open it in your favorite Javascript IDE. Lets start off with including the packages that we’ll need throughout the code. We do this with the require method, like so:

const fetch = require(‘node-fetch’);

const fs = require(‘fs’);

As mentioned above, node-fetch package will help us manage HTTP requests in an easier fashion than using Node.js’ native XMLHttpRequest. While fs is a super easy-to-use native Node.js package which enables interaction with the file system.

At this point we can go ahead and build the URL skeleton that we’ll be using to request our data from the API. In this example we’re calling an API that provides stock price data.

var _URL = {

authority: ‘https://’,

host: 'cloud.iexapis.com',

endpoint: '/stable/stock/',

quote: 'aapl',

parameters: '/quote?',

api: 'token=[PLACE YOUR TOKEN HERE]'

};

Separating the URL in different parts as done above has its advantages. For example, if we wanted to iterate different stock prices from an array, we’d simply attribute a variable to the ‘quote’ property and execute asynchronous API calls. Along the following lines:

for(i=0; i < quoteList.length; i++){

url = _URL.authority +

_URL.host +

_URL.endpoint +

quoteList[i] +

_URL.parameters +

_URL.api;

urls.push(url)

}

Great, we’re almost there! Now we can call the fetch function to retrieve our data so we can chew and digest it.

Retrieving the data from the API

Below is an example of how we could go about retrieving a single fetch and discerning what is important for our application:

fetch(url)

.then(response => {

return response.json();

})

.then(data => {

Object.keys(data).forEach( key => {

console.log(`${key} : ${data[key]}`);

});

})

.catch(err => {

console.log(err);

});

Lets break down the above code in pieces and check out what’s happening.

fetch(url)

.then(response => {

return response.json();

})

In this piece the fetch method makes a HTTP request to the described ‘url’ address. The method returns a promise containing a response object which will be (hopefully) populated with the data we require. We call the json() method from the response object, which is a shortcut to getting our data parsed from the body property. And finally we return the result to the next callback function.

.then(data => {

Object.keys(data).forEach( key => {

console.log(`${key} : ${data[key]}`);

});

})

In this piece we get the keys of the JSON object returned by the previous function block, and iterate through to get a list of all key — value pairs. Our output will be similar to the following:

symbol : AAPL

companyName : Apple Inc

primaryExchange : NASDAQ/NGS (GLOBAL SELECT MARKET)

latestPrice : 126.655

latestSource : Close

latestTime : December 18, 2020

latestUpdate : 1608325201311

At this point, we can get the data we require and continue to process according to our application’s objectives. Hopefully this will get you started with Node.js API calls.

Until next time, thanks for reading!
Jackson @JsonView and it’s meaningful use with Spring Boot REST
By: Amit Patil
Tags: Rest Api, Spring, Spring Boot, Java, Json

Most of the time we come across requirements in an application where we want to provide different views of the same model data to different clients/users.

To solve this, we may end up creating multiple models/DTOs to represent different views of the same state to different clients. For models with a limited number of fields this approach may work, but for large models, it will become difficult to manage and will result in duplication of code.

Jackson’s @JsonView comes to the rescue

We can use Jackson’s @JsonView to solve the above problem. Similar to database views, where we can build multiple virtual tables(views) with different combinations of columns in the underlying table, we can define multiple views of the same Model/DTOs with different combinations of fields.

How to use? 🤔

Define view as a class or interface. Use @JsonView annotation in Model/DTOs to map fields to one or more views.

Jackson reads these @JsonView annotations on fields during serialization/ de-serialization of objects and serialize/deserialize only fields in view and will skip all other fields.
JSON as an alternative to XML in AJAX / Web Services Requests
By: Damely Tineo
Tags: Ajax, Stringify, Xml, Web Services, Json

JSON as an alternative to XML in AJAX / Web Services Requests Damely Tineo Jan 22·3 min read

JSON (JavaScript Object Notation) “is a lightweight data-interchange format.” — json.org

JSON is used primarily to receive and transmit data from a server and is characterized by key-values/objects. Keys being a string enclosed in quotation marks and values: strings, numbers, booleans, arrays, or objects.

The JSON syntax is built on two structures: arrays and objects. Objects which can also contain arrays or key-value pairs (as mentioned above):

[{

"name": "Biscuit",

"age": 7,

"breed": "frenchie",

"owner": {

"name": "Daisy",

"homeTown": "Philadelphia"

...

},

"hobbies": [

sleeping,

eating,

...

],

},

{

"name": "Lola",

"age": 2,

"breed": "chihuahua",

...

}] Above see a collection of JSON objects, these are specific dog objects within an array of dogs.

Pretty much every modern programming language has a way to convert a JSON string into an object it can work with. Even our browsers can do this. It’s extremely important, however, that when writing JSON we implement and follow the correct syntax rules and format in order to be able to properly parse the data. Remember that computers and software can’t read minds…well at least not yet!

Where is JSON used?

Web services

As the name implies, in JavaScript!

Note: While it derives from JavaScript, JSON is language-independent, many programming languages include code to generate and parse JSON-format data therefore it is also supported by most backend technologies.
Eureka client in Kotlin
By: Mohammed Hewedy
Tags: Parsing, Eureka, Serialization, Kotlin, Json

JSON Serializer

I have a history of traversing object graphs and converting them from one shape to another, so I decided to start wit the Serializer.

Here’s the starting function of the serliazer:

fun serialize(obj: Any?, ignoreNull: Boolean): String? = obj?.let {



val jsonStr = callToJsonFunction(obj)

if (jsonStr != null)

return@let jsonStr



val properties = obj::class.members

.filter { isProperty(obj, it) }



if (properties.isNotEmpty()) {

return@let properties

.map { serialize(obj, it, ignoreNull) }

.filter { !ignoreNull || it != null }

.filterNotNull()

.joinToString(prefix = "{", separator = ",", postfix = "}")

}



if (obj is Enum<*>) { // enum with no properties

return@let obj.name.doubleQuote()

}



return@let null

}

The code is not so straightforward. however, it checks first if the object has a function called toJson and if so, it calls it and returns. so it allows the user to override the serialization behavior for specific objects.

If not, it starts to iterate over object properties and serialize each one of them. it calls another serialize method to do so.

Eventually, it put the result of each call to the function seralize on each property in the form of {<propertyName>: <result of serialize call>, ...}

Here’s the implementation of the function serliaze :

fun serialize(obj: Any?, property: KCallable<*>, ignoreNull: Boolean): String? {

val value = when {

property.returnType in numberTypes -> property.call(obj)

property.returnType in stringType -> (property.call(obj) as String?)?.doubleQuote()

property.returnType.arguments.isNotEmpty() // line 7

&& isCollection(property.returnType) -> serializeCollection(property, obj, ignoreNull)

else -> serialize(property.call(obj), ignoreNull) // line 10

}



return if (ignoreNull && value == null) null

else property.name.doubleQuote() + ":" + value

}

And this function use pattern matching on the property type, and distinguish between number types, string types, collections and finally regular objects.

In the case of numbers, it spits the number as it is, and in case of strings, it wraps it in double-quotes.

In the case of objects, it recalls the original serialize method that accepts an object (not the one that accepts the property)(line 10 above)

However, in the case of collections (line 7), it calls the function serializeCollection which iterates over elements and then call the original serialize method again (that accepts the object), and wrap results between JSON array brackets “[“ and “]”.

Full source code for the serializer is here: https://github.com/mhewedy/eureka-klient/blob/master/src/main/kotlin/helpers/json/Serializer.kt
Python Read/Write in Json File
By: Miss Discontinuity
Tags: 这就是街舞, Json, Python, Wang Yibo

python day 33

Python and Java are two popular programming languages.

Today we will use This is dance as an example to write a small program to read and write Json files in Python.

JSON (JavaScript Object Notation) file is a lightweight, text-based editor that stores simple data structures and objects, and it is a standard data interchange format. It is primarily used for transmitting data between a web application and a server.

Python Code:

import json

import codecs def dance():

mydict = {

"show": "Street Dance of China 这就是街舞 ",

"season": 3,

"time": 2020,

"Leaders": ["王一博 Yibo Wang", "张艺兴 Lay Zhang", "王嘉尔 Jackson Wang", "钟汉良 wa Zhong"],

"Dance categories": [

{"Category": "Old school", "Star": "亮亮 Spark hhl"},

{"Category": "Locking", "Star": "韩宇 Han yu"},

{"Category": "Wacking", "Star": "AC 小公主"},

{"Category": "B-BOY", "Star": "蛇男 She nan"},

{"Category": "Urban", "Star": "余衍林 Franklin Yu"},

{"Category": "Jazz", "Star": "淡淡 Dan dan"}

]

}

print(json.dumps((mydict), indent=2, ensure_ascii=False)) #save json files

# try:

# with open('data.json', 'w', encoding='utf-8') as fs:

# json.dump(mydict, fs)

#except IOError as e:

# print(e)

#print('Saved! ')



#import json file

#import requests

#import json #def main():

# resp = requests.get('http://api.tianapi.com/guonei/?key=APIKey&num=10')

# data_model = json.loads(resp.text)

# for news in data_model['newslist']:

# print(news['title']) #main() dance()

Json Code:

{

"show": "Street Dance of China 这就是街舞 ",

"season": 3,

"time": 2020,

"Leaders": [

"王一博 Yibo Wang",

"张艺兴 Lay Zhang",

"王嘉尔 Jackson Wang",

"钟汉良 wa Zhong"

],

"Dance categories": [

{

"Category": "Old school",

"Star": "亮亮 Spark hhl"

},

{

"Category": "Locking",

"Star": "韩宇 Han yu"

},

{

"Category": "Wacking",

"Star": "AC 小公主"

},

{

"Category": "B-BOY",

"Star": "蛇男 She nan"

},

{

"Category": "Urban",

"Star": "余衍林 Franklin Yu"

},

{

"Category": "Jazz",

"Star": "淡淡 Dan dan"

}

]

}

happy studying!
Installing drogon HTTP web server
Tags: Cpp, Cpp17, Json, Rest Api

While researching a REST server framework in C++ 17, we came across the Drogon project. We tried to install it on MacOS following the instructions described in https://github.com/an-tao/drogon/wiki/installation:



$ cd drogon

$ git submodule update --init

$ mkdir build

$ cmake .. -B .

$ make $ git clone https://github.com/an-tao/drogon $ cd drogon$ git submodule update --init$ mkdir build$ cmake .. -B .$ make

Which led to the following errors:

[ 49%] Linking CXX executable webapp_test Undefined symbols for architecture x86_64: "Json::writeString[abi:cxx11](Json::StreamWriter::Factory const&, Json::Value const&)", referenced from: ... "Json::StreamWriterBuilder::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)", referenced from: ... ld: symbol(s) not found for architecture x86_64

collect2: error: ld returned 1 exit status

make[2]: *** [examples/webapp_test] Error 1

make[1]: *** [examples/CMakeFiles/webapp_test.dir/all] Error 2

make: *** [all] Error 2

To fix the issue, jsoncpp need to be (re)installed. If it is already installed, uninstall it using: brew uninstall jsoncpp .

Then install it from the source. To build it, you’d need ossp-uuid , meson , ninja : brew install ossp-uuid meson ninja `

Then to build it:



$ cd jsoncpp

$ BUILD_TYPE=debug

LIB_TYPE=shared

meson --buildtype ${BUILD_TYPE} --default-library ${LIB_TYPE} . build-${LIB_TYPE}

ninja -v -C build-${LIB_TYPE}

cd build-${LIB_TYPE}

meson test --no-rebuild --print-errorlogs

sudo ninja install $ git clone https://github.com/open-source-parsers/jsoncpp $ cd jsoncpp$ BUILD_TYPE=debugLIB_TYPE=sharedmeson --buildtype ${BUILD_TYPE} --default-library ${LIB_TYPE} . build-${LIB_TYPE}ninja -v -C build-${LIB_TYPE}cd build-${LIB_TYPE}meson test --no-rebuild --print-errorlogssudo ninja install

Then go back to drogon/build and rerun make .

To test the examples:

$ cd examples

$ ./webapp

20190216 21:43:38.066290 UTC 271689 DEBUG [TestController] TestController constructor - TestController.h:20

_

__| |_ __ ___ __ _ ___ _ __

/ _` | '__/ _ \ / _` |/ _ \| '_ \

| (_| | | | (_) | (_| | (_) | | | |

\__,_|_| \___/ \__, |\___/|_| |_|

|___/ 20190216 21:43:38.077896 UTC 271689 INFO Start to run... - HttpAppFrameworkImpl.cc:200

20190216 21:43:38.079254 UTC 271689 WARN HttpServer[drogon] starts listenning on 0.0.0.0:8848 - HttpServer.cc:70

In another terminal:
Flutter & JSON
By: Petrus C Devilliers
Tags: Flutter, Restful Api, Flutter App Development, Flutter Web, Json

We’ll be covering the recommended way for smaller projects to handle JSON Data, which happens in the project’s code. We’ll have to manually add the handlers for any of our classes that we want to be able to initialize from JSON data. First, an example class:

class User {

int id;

DateTime createdAt;

String name;

int age; User({this.id, this.createdAt, this.name, this.age});

}

Mock API:

In this example we’ll be using the data returned from mockapi.io, where you can sign up for free in order to mock data returned from a Restful API. The data returned from our mock API looks like this:

[

{

"id": "1",

"createdAt": 1604386144,

"Name": "Name 1",

"Age": 24

},

{

"id": "2",

"createdAt": 1604386084,

"Name": "Name 2",

"Age": 87

},

{

"id": "3",

"createdAt": 1604386024,

"Name": "Name 3",

"Age": 15

},

{

"id": "4",

"createdAt": 1604385964,

"Name": "Name 4",

"Age": 87

},

{

"id": "5",

"createdAt": 1604385904,

"Name": "Name 5",

"Age": 71

},

{

"id": "6",

"createdAt": 1604385844,

"Name": "Name 6",

"Age": 24

},

{

"id": "7",

"createdAt": 1604385784,

"Name": "Name 7",

"Age": 8

},

{

"id": "8",

"createdAt": 1604385724,

"Name": "Name 8",

"Age": 47

}

]

Maps and you:

The data type a JSON string is converted to is a Map, which consists of Key Value pairs. The Key is what we use to access the Value in the Map, for example:

Map<String,dynamic> example = {"name":"Bob","age":27,"height":1.8}; print(example["name"]);

Will print the value “Bob”. If you want to get down to the nitty gritty you can find the documentation on the Map class here.

Maps and our class:

So back to our User class, first we’ll need to import dart:convert, next we’ll be adding a constructor to the class which accepts a Map<String, dynamic> parameter, this Map uses the names of the JSON variables as the Key to access their Values. The value part of the Map is specified to be dynamic as we may receive many different data types from a single JSON object. If you are certain that you will only ever receive one type of data from your JSON object you could specify the Map to store values of that type, for instance: Map<String, int> would be used if the only type of members stored in your object are integers.

class User {

int id;

DateTime createdAt;

String name;

int age; User({this.id, this.createdAt, this.name, this.age}); User.fromJson(Map<String, dynamic> json){

id = json['id'];

createdAt = DateTime(json['createdAt']);

name = json['name'];

age = json['age'];

};

}

Remember to check your spelling as the names of the keys are case sensitive. Once you’ve done everything correctly, you should be able to then create an instance of our User class from JSON data like this:
“Any” is the Nero of your Typescript codebase
By: Ashish Kirodian
Tags: How To, Type, Practice, Typescript, Json

Nero watching Rome as it burns from the fire he started.

Just like Nero burnt Rome and then watched it from distance, you will be allowing Any to do the same to your codebase. Except for the fact that Nero did it for the revamp of administration, while Any will ruin your codebase to the core.

When I say Ruin, you will basically end up writing JS code instead of Typescript even if you are using Typescript as the language.

I have noticed some codebases and stack overflow questions till now and confidently say, there are some “developers” who saturate the core goal of the Typescript language by declaring or casting almost everything into Any.

So, why use Typescript at all if you are going to saturate the types? I still wonder what inspires them to do so.

Maybe it is because they don't want to spend time setting up architecture for the application they build? Or they are too lazy to just Hover over the variable in the VS Code and determine its type? Or they don't completely understand Typescript? Lets not judge :p

I've seen many developers who write React classes or functions just to replicate the design/wireframe that is given to them. They never think of a long term architecture. Scalability is just another word for them.

The irony is that they even typecast the events that are fired from an event listener to Any. Just because they couldn't deal with understanding what type to be made compatible with the code they are writing.

I have some tips to follow so that you don't saturate your codebase.
Using New York Times API and jq to collect news data
By: Dana Lindquist
Tags: Jq, New York Times, Json, API, Data

Are you looking for news data? The New York Times API (Application Programming Interface) is the place to go. They have an API that can be accessed free for non-commercial uses. In this article I will explain how to get this data.

Create an account and get an API key

The first thing to do is go to the NY Times Developer web site (developer.nytimes.com) and create an account. They have a great section called “Get Started” which will walk you through this process. Using their API is free for non-commercial use.

You will need an API key which they use to monitor usage levels. There is a limit to the number of requests you can perform in a day but I never ran into the limit.

As described in the “Get Started”, create an App and choose which APIs you would like to access with that App. I’m going to describe how to access the Archive API. Once you choose CREATE for the App you will be given an API key.

Use requests in Python to pull the json files

An Archive API request will pull all documents from a given month. The years can be from 1851 to 2019 and the months from 1 to 12. In the example shown here the Archive is pulled for December (month 12) in 2018.
Jolt Transformation Overview
By: Saurav Daruka
Tags: Json Transformation, Json Jolt Transformation, Json Json Transformation, Jolt, Json

Jolt (JsOn Language for Transform) is a transformation library, written in Java that allows a developer to convert one JSON structure to another. Jolt provides a set of transformation types, each with their own DSL (called specifications), that define the new structure for outgoing JSON data.

Why Jolt?

JSON is a language independent data format, commonly used for client-server communication especially when it comes to web-services, configuration-files, other data-descriptors, etc. The incoming JSON data simply needs to be formatted or re-labeled for use in another system or repository, such as Hive, HBase, MongoDB or Elasticsearch.

Before Jolt, input JSON had to go through various conversions before getting the JSON output:

Input JSON -> XML -> XSLT/STX -> XML -> JSON Output

After Jolt, only the Jolt specification file (in JSON format) needs to be written and applied to input JSON to get the JSON output.

Input JSON -> spec.json -> JSON Output

Also with Jolt, one can gain below advantages:

It minimizes Java code changes as most of the work is done using transformations.

Any complex transformation logic which can’t be expressed in standard terms can be plugged in via a Java extension class with Jolt.

It avoids having to call hundreds of getters and setters to move from one object model to another, which makes moving from one service’s API to an internal model or an internal model to an external format much easier.

Jolt Transformations:

Jolt allows the below transformations to generate the desired result:

shift : copies data from input to the output tree

default : applies default values to the tree.

remove : removes data from the tree.

sort : sort the Map key values alphabetically.

cardinality : adjusts the cardinality of input data.

Example using Shift operation:

Input JSON:

{

"restaurant": {

"rating": {

"value": 3

},

"address": {

"value": "India"

}

}

}

Jolt Spec:

[

{

"operation": "shift",

"spec": {

"restaurant": {

"rating": {

"value": "Restaurant Rating"

}

}

}

}

]

Output JSON:

{

"Restaurant Rating" : 3

}

Useful Resources:
Spring Boot — Dynamically ignore fields while serializing Java Object to JSON
By: Vicky Av
Tags: Spring Boot, Json, Jackson

Lets consider we have a RestController class which has two endpoints to expose User & List<User>

User & UserController class

Since id & dob fields in User object are sensitive, we don’t want to expose them in our endpoints.

Jackson library provides @JsonIgnore & @JsonIgnoreProperties annotations to ignore fields while serializing

Here is a sample code

One catch with @JsonIgnore & @JsonIgnoreProperties annotations are — They expect field names in compile time itself. But what if our consumer want to ignore few fields dynamically based on their input ?

We can use MappingJacksonValue class to achieve the above requirement

Here is the sample code

Sample which uses MappingJacksonValue

Remember the below points

Properties which are to be ignored is supplied to SimpleBeanPropertyFilter.serializeAllExcept(“id”, “dob” ). This can be easily made dynamic based on user input

). This can be easily made dynamic based on user input Filter name mentioned in SimpleFilterProvider() .addFilter(“ userFilter ”, simpleBeanPropertyFilter) has to match filer name provided in @JsonFilter( “userFilter” ) annotation on User class

”, simpleBeanPropertyFilter) has to match filer name provided in @JsonFilter( ) annotation on User class We need to return MappingJacksonValue instance instead of User or List<User> (Refer line 19 & line 34)

Thats all folks !

Follow me on Twitter - @IamVickyAV
Masashi, thank you for this in-depth and INCREDIBLY helpful article!
By: Kevin Howard
Tags: C Sharp Programming, Webgl, Debugging, Unity, Json

Masashi, thank you for this in-depth and INCREDIBLY helpful article! It is the best I have found on migrating to WebGL in Unity.

Your note below was exactly where our project was stuck:

Console log says something like “constructor is not found for Invoice class”.But, We had a default constructor and Json annotation and etc. but not working with this error. We were devastated with no clues to fix this issue. After some days,…

Our project is using WebGL with a large set of .Net Standard 2.0 libraries that are intentionally Unity-free. Our .Net Standard 2.0 libraries utilize Newtonsoft.Json extensively. We started to get exactly the same “constructor not found” type error, and I was pulling my hair out for days trying to find ways to work around it. I was concerned we would have to do a major rewrite of many of the critical .Net Standard libraries our project needed. We were completely stuck. Your comment about “we were devastated” really resonated with me.

But your find on IL2CPP stripping was the unlock!

I was able to solve the stripping issue in a slightly different way, however. Following the instructions at https://docs.unity3d.com/ScriptReference/Scripting.PreserveAttribute.html I added a new preserve attribute to our .Net Standard libraries:

[System.AttributeUsage(System.AttributeTargets.Method | System.AttributeTargets.Constructor)]

public class PreserveAttribute : System.Attribute

{

}

I then marked our JSON constructors with this new attribute, like this:

[PreserveAttribute]

[JsonConstructor]

public PlayerData(string name="", int value=0)

{

...

}

This solved the IL2CPP stripping and Newtonsoft JSON constructor issues for us in WebGL builds.

Also, your advice on “How to debug WebGL build” was great guidance. “All we could do was the good old naive logging lines onto Web browser console” is a great start and has been very helpful. We added a logging class that has multiple log levels (Logger.LogInfo, Logger.LogWarning, Logger.LogError, etc.). We use the info logs extensively in our Unity code so whenever an error happens we can quickly find it. We can then turn off info level logs with a feature flag for production release builds.

An additional assistance has been turning on Unity 2019’s Player Settings -> Publishing Settings -> Full With Stacktrace and Debug Settings during development. This gives clear indictment of the place where your code fails during WebGL development.

Masashi, thank you again for the great article and advice!
Parsing data with Jackson using kotlin
By: Pavneet Singh
Tags: Parsing, Jackson, Kotlin, Programming, Json

Parsing with Jackson and Kotlin

Jackson is one of the famous library to parse XML or JSON data though with Kotlin, some key factors are required to avoid unexpected issues.

Need of Default Constructor

By default, Java provides a default constructor(if there’s no parameterized constructor) which is used by Jackson to parse the response into POJO or bean classes.

Kotlin provides a feature of data classes which do not provide a default constructor(instead, it uses a parameterise constructor) so parsing a response in Kotlin will result in InvalidDefinitionException exception as:

where the Person class is a simple data class with two properties

data class Person(var name: String, var occupation: String)

and the exception log.

Exception in thread "main" com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `Person` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)

at [Source: (String)"{

"name": "Pavneet",

"occupation": "Software Engineer"

}"

Data classes has a restriction to have at-least one primary constructor parameter so no default/no-args constructor is available by default in data classes.

Jackson is unable to find the default constructor so it’s unable to create an object of Person class and throwing the InvalidDefinitionException exception.

This issue can be fixed by using @JsonProperty annotation which will instruct Jackson to use setters and getter for parsing as:

The exception can also be fixed by creating a normal class instead of data class as:
System.text.json VS Newtonsoft.json
By: Sami C.
Tags: Newtonsoft, Json, Net Core Serializer

Microsoft shipped a new namespace called System.Text.Json with .NET Core 3.0. This one is integrated with .NET Core 3.0, so you no longer need to install a nuget package.

Why a new JSON serializer?

From what I’ve read I can basically tell you that Microsoft had trouble packaging Newtonsoft inside .NET Core 3.0. So they had to come up with a middle ground which ended up to be system.text.json. There’s also a speed improvement between the new one and the old one. Unfortunately, at a cost of some features that newtonsoft has.

Let’s take a look!

What is the new JSON serializer missing?

So I tried it on an existing code base and ended up with some interesting results:

The new json serializer does not support datamember attributes for naming

[DataMember]

It can be a lot of effort to change all your DTO’s to support the new serializer.

Here is the original: https://github.com/dotnet/runtime/issues/30009

Speed

If you really want speed, then the new JSON serializer is most definitely not your best option. Utf8Json and JIL have been proved to be 2–4 times faster.

Referential loop handling

This sample sets ReferenceLoopHandling to Ignore so that looping values are excluded from serialization instead of throwing an exception.

string json = JsonConvert.SerializeObject(foo, Formatting.Indented, new JsonSerializerSettings

{

ReferenceLoopHandling = ReferenceLoopHandling.Ignore

});

This is not supported in the new JSON serializer, you’ll end up with exceptions. Although Jozkee made a comment that it should be available in the next version of .NET Core, which will be .NET 5.

Take a look at the comment:

More information can be found here: https://github.com/dotnet/runtime/issues/29900

Serializing dynamic types

The new serializer cannot serialize dynamic types.

dynamic p = JsonSerializer.Parse(json, typeof(ExpandoObject));

Github issue: https://github.com/dotnet/runtime/issues/29690

Conclusion

Let’s do a quick summary:

You want things to work and not worry about missing features?

Newtonsoft is the one to pick here!

Speed is your top priority?

Use something like Utf8Json.

Starting a new project?

I would use System.Text.Json and try to work around the missing features for now. I definitely believe there will be some improvements in the next version of .NET Core. Having the new serializer in your new projects will definitely give you a step ahead when upgrading to .NET 5.

Existing projects?

I advise you to stay on Newtonsoft until the missing features are actually added.

I hope you enjoyed the breakdown between the two.

Happy coding!
5 places we can use JSON.stringify and JSON.parse
By: Jayanth Babu
Tags: JavaScript, Technology, Programming, Json, API

JSON (JavaScript Object Notation) is a general format to represent data as objects and mimics the object format from JavaScript. Initially, it was made only for JavaScript but many other languages like Java, Python, and Go are using this format for sharing the data between client and server.

Let's discuss some real use cases.

Storing and retrieving data from localStorage or sessionStorage

localStorage and sessionStorage properties allow us to save key/value pairs in a web browser.

Local storage can only save strings, so storing objects requires that they be converted into strings using JSON.stringify before storing them. You can’t ask local storage to store an object directly because it will store “[object Object]”.

Also, we need to run JSON.parse when retrieving the data from local storage. Let's see the example below:

XMLHttpRequest to fetch the data from a server

When retrieving the response from an API, by default the response comes in the string format, and we need to parse it into JSON.

Stringify the body when making a POST request

We need to stringify the body object when sending a POST request to the server. Let’s see the example below.

Deep cloning a JavaScript object

Copying objects in JavaScript can be tricky. There are different ways to perform a shallow copy, which is the default behaviour in most of the cases. However, it can be much more challenging to create a deep copy of an object.

By using JSON.stringify , we can create a deep clone of an object in JavaScript.

In JSON, functions are not allowed as object values. JSON.stringify() will remove any functions from a JavaScript object, both the key and the value. Thus, this deep clone is for data properties only.

The JSON.stringify() function is supported in all major browsers.

Convert any data type into a string

We can convert any kind of data type value into a string by using JSON.stringify .

While developing an application using JavaScript, many times it is needed to serialize the data to strings for storing the data into a database or for sending the data to an API. The data has to be in the form of strings.

That's it! I hope this article helped you increase your understanding of JSON.

Thanks for reading :)
Customizing Protobuf JSON Serialization in Go
By: Sebastian Nyberg
Tags: Go, Json, Golang, Protobuf

A new version of Protobuf was released in March 2020, See this post for more info. Please ensure that you have the latest version of the protoc-gen-go plugin, or this article may diff from what you see locally.

Let’s start with a toy example:

Generate the code:

protoc --go_out=. todo.proto

Output:

Create and serialize an example Todo:

JSON output:

There are two obvious problems with this: the enum is serialized as an int, and the timestamp is serialized as.. a struct of seconds and nanos?

protocolbuffers/protobuf-go/encoding/protojson

The Golang JSON serializer doesn’t deal well with Protobuf. Instead, you should use protojson:

The output is now:

The timestamp is now in RFC3339 format, and the fields are camelCased.

protojson Marshaler Options

Let’s look at the protojson.Marshal function:

As you can see, proto.Marshal is just a convenience function which initializes MarshalOptions with default settings.

MarshalOptions has a number of dials:

Multiline and Indent — print the message on multiple lines, using the provided indent.

UseProtoNames —Effectively makes the JSON keys snake_cased instead of camelCased.

EmitUnpopulated — Explicitly include unpopulated values in the output (see next section)

UseEnumNumbers — print enums as ints instead of their .String() representations. This is what the regular JSON marshaler gave us earlier.

MarshalOptions.EmitUnpopulated

EmitUnpopulated can be set to include default values in the response. This is very helpful when default values carry meaning and I would advice to let consumers manage default values rather than the API omitting them.

Let’s say we want to keep track of how many times users have viewed a todo. We add a new view_count field to the Protobuf definition:

If ViewCount is unset or set to zero, then the resulting JSON will be just as before:

In this case, it would make sense to include that view count is zero.

To remedy this issue, set EmitUnpopulated and try again:

MarshalOptions.UseProtoNames

UseProtoNames changes the output to use the Proto names, which are snake_cased:

MarshalOptions.Resolver
Working with compressed JSON in Python
By: R A
Tags: Zip, Json, Python, Compression

A long JSON file can contain many redundancies, especially repeating keys.

For example, the real-time bus GPS data for Taipei during daytime contains hundreds of records in the following kind:

The whole unformatted file is 1'260'575 bytes long. Zipping the file results in an archive of 78'478 bytes. Thus, even before dumping the whole JSON record to disk, it makes sense to compress it. We will discuss how to do that in this note. The code is available here and here.

Compression

We first discuss the compression procedure in the interactive Python shell. We will use the following packages:

>>> import zlib, json

>>> from base64 import b64encode, b64decode

Let’s start with a JSON object, like this

>>> original = {'a': "A", 'b': "B"}

This is the object to compress:

>>> original

{'a': 'A', 'b': 'B'}

First, serialize the original object into a string:

>>> json.dumps(_)

'{"a": "A", "b": "B"}'

Next, encode it as a bytes object:

>>> _.encode('utf-8')

b'{"a": "A", "b": "B"}'

Compress the bytes into another bytes object:

>>> zlib.compress(_)

b'x\x9c\xabVJT\xb2RPrT\xd2QPJ\x02\xb1\x9c\x94j\x01-\xea\x04O'

Shrink the character range to 64 human-readable characters for easier inspection and handling:

>>> b64encode(_)

b'eJyrVkpUslJQclTSUVBKArGclGoBLeoETw=='

Interpret the bytes object as a string:

>>> _.decode('ascii')

'eJyrVkpUslJQclTSUVBKArGclGoBLeoETw=='

Package the result in a dictionary to include some mnemonic meta-info about the object:

>>> {'base64(zip(o))': _}

{'base64(zip(o))': 'eJyrVkpUslJQclTSUVBKArGclGoBLeoETw=='}

Coming back to the real-time bus data, the size of the compressed object is down to 155kb bytes from 1.26Mb. This is twice the size of the brute-zipped file because we used here the base64 encoding, but still an order of magnitude smaller than the original.

This is the whole compression procedure in a function:

Decompression

The reverse process of decompression is then:

>>> json.loads(zlib.decompress(b64decode(_['base64(zip(o))'])))

{'a': 'A', 'b': 'B'}

Note that

the compression works on any object serializable by json.dumps , not just JSON objects;

, not just JSON objects; the decompressed object may differ from the original one, for example json.loads(json.dumps((1, 2))) returns the list [1, 2] instead of the original tuple (1, 2) .

In some instances, however, we may want to admit compressed or uncompressed input, as the case may be. Therefore, in the decompression function we first check if the input is in the compressed format. If it is not, a RuntimeError is thrown — unless this error is explicitly suppressed by passing insist=False and the input object is returned unmodified.

JavaScript

To compress/decompress client-side check out this StackOverflow post.

The complete python module with unit tests
Fetch Data Using JSON API And Insert Into SQLite3 DB
By: Tejas Mokashi
Tags: Sqlite3, Python, Insert, Json, API

Import below mentioned libraries to perform the operation

Module Info :

1 ) requests : Using requests you can add content like headers, form data, multipart files, and parameters via simple Python libraries.

2) json : Json used to Encode and Decode the json data.

3)datetime : This module used for convert datetime in python’s understandable format.

Fetch json data threw API by hitting on Quandl website

url = “https://www.quandl.com/api/v3/datatables/SHARADAR/SFP.json?api_key=sCJNpBm_3CDYKxgRbMFS"

content = requests.get(url).content

dataset = json.loads(content)

Create list for append the data

dataframe=list()

Filter columns for fetching data in for loop

Use for loop to fetch data from dataset.Here I used datetime module’s strptime() function to convert the date into python understandable format.after converting date I used to store all data in data object.and then append data in list which I created above.

I received data in dataset in list format so i used for loop to perform operation through the list and convert 1st and 9th column in date format.other columns converted in float.and used append() function to insert data in dataframe.
Kotlinx Serialization
By: Egbai Smile Mmumene
Tags: Kotlin Native, Android, Kotlin, AndroidDev, Json

Setting up your project to use the plugin is quiet simple, but I will include a complete setup for both android projects as well as multiplatform projects. But feel free to check out the Kotlinx.Serializer repo for instructions on this and more.

One small hiccup on the multiplatform serialisation setup is the need to add this dependency under the native dependecy section of your gradle script.

implementation org.jetbrains.kotlinx:kotlinx-serialization-runtime-native:0.9.1

SETUP GRADLE EXAMPLE

For multiplatform:

For Android:

Serialization

Serializing is done quite easily, for one you would need to annotate the intended class with the @Serializable annotation as below:

Note: This can as well work with data classes .

Now to actually use this in action let’s take an example of how to convert a JSON to object and back

The ease and simplicity which the kotlinx serializer helps us to convert objects in a clean way is absolutely amazing.

Two more annotations to note are transient and optional. Using transient will have the serializer ignore that field and using optional will allow the serializer not to break if a field is missing, but at the same time a default value will need to be provided.

Android Use Case for Retrofit

For those who would like to as well use this library in android, Retrofit 2 has an adapter for converting JSON responses to objects. The adapter can be found here. Some Sample code of adding the adapter

Assuming your class already has the required annotations, after making a request this should serialise your class into the required object.

All in all, the kotlin serializer is a wonderful addition to any project and makes serialization a painless process when you want to save objects as a string or parse them into other data structures. Here is a sample repo for the android setup with a small(non-architectural) working example:

Here is the official Serializer repo and as well the retrofit adapter
GoLang : Dynamic JSON Parsing using empty Interface and without Struct in Go Language
By: Mohammed Irshad
Tags: Parsing, Golang, Golang Tutorial, Json, Empty Interface

4. Parsing Embedded object in Array of JSON

Output:

Reading Value for Key : 0

Id : 1 - Name : Mr. Boss - Department : - Designation : Director

Address : Mumbai Maharashtra India

Reading Value for Key : 1

Id : 11 - Name : Irshad - Department : IT - Designation : Product Manager

Address : Mumbai Maharashtra India

Reading Value for Key : 2

Id : 12 - Name : Pankaj - Department : IT - Designation : Team Lead

Address : Pune Maharashtra India

Try it at : https://play.golang.org/p/P54UcFy4uJt

Code Explanation: /* First: declared array map of string with empty interface

which will hold the value of the parsed json. */

var results []map[string]interface{} /* Second: Unmarshal the json string string by converting

it to byte into map */

json.Unmarshal([]byte(empArray), &results) /* Third: Read the each item from an array of map using range */

for key, result := range results {} /* Fourth: To Read data from address node from each item we have to declare one more map with empty interface inside range loop */

address := result["address"].(map[string]interface{}) /* Fifth: Read the result value by its key inside range loop */

fmt.Println("Id :", result["id"]) /* Sixth: Read the address value by its key inside range loop */

fmt.Println("City :", address["city"])

Parsing JSON in go is very each and its depends upon developer preferences whether they want to use struct or they want to parse them raw using interfaces.

Will keep posting new articles! Thank you!!!
iOS + Timezone Handling
Tags: Timezone, iOS, Json, Dateformatter

This article is about timezones and how it has to be handled to avoid date and timezone conflicts on a different region.

Mobile apps and their development heavily rely on servers data. Where we most often have 2 variables startDate and endDate . The backend server could have stored this variable as String or Int64 (long). And here’s where the problem starts, nothing but a Timezone.

UTC (Coordinated Universal Time), a standard for setting time zones in the world. It is used to determine the time in the different zones. Time zones in the world are expressed by their +ve or -ve Offsets relative to UTC.

For eg: India is in the IST zone, i.e. UTC + 5.30, which means that if you are in India and it is 11 am (IST = UTC + 5.30), according to UTC (UTC = IST — 5.30) it is now 5.30 am.

ISSUE

we came across a situation where the backend server stored startDate and endDate in the following String format.

Obviously, mobile apps use DateFormatter API to convert the string into a date object. String to Date conversation was working fine for a region where the date was updated. But it was failing in other regions since the server timezone information (.247Z) was not matched with the current region's Timezone. Here we can see that JSONSerialization was failing due to the “Date string does not match the format expected by formatter.” We have addressed and fixed those issues using FIX 2.

# FIX 1

Timezone issues can be avoided by storing proper UTC Milliseconds (Int64 or Long) in startDate and endDate properties. but the mobile app should make sure that it always sends the proper UTC Milliseconds to the server, failing to send can pollute the server data.

# FIX 2

Before sending data to the server, mobile apps can convert the date object into a UTC string which internally strips down the timezone information. The following util can help in doing such conversation.

Sending data to the server

On the other hand, While receiving the data from the server, the date string can be converted into a proper date object by the following extension.

Conclusion

Once after the proper conversation, we can use the date object to paint our user interface by adding timezone offsets if needed.

When using dates in mobile applications, we have to be very careful when sending them to the server. In order to use time which takes the time zone into account, remember to send it in UTC format, and then convert it to your local time zone upon receipt.
Making API requests in flutter
By: Antony Ruban
Tags: Flutter, Dart, API, Json, App Development

API call is one of the most common functions that are performed in mobile applications.Flutter itself provides tools to make http requests with http package and also there are other packages like Dio. Dio is a powerful Http client for Dart, which supports Interceptors, Global configuration, FormData, Request Cancellation, File downloading.

Nowadays Most of the apps use api to connect to the backend and it’s also much safer to place the important logic of the application in the server and fetch it using api instead of coding directly in the mobile app.We’ll see in this post on how to make a get request to the server.

Add the http Package:

Install the http package by adding it to the dependency section in the pubspec.yaml file.The latest version as of writing this is 0.12.0+3.You can find the package here.

dependencies:

http: ^0.12.0+3

Run flutter packages get to download the package and import the package in your dart code using the below code.

import 'package:http/http.dart';

Make the request:

We need to make the request to the server to fetch data and use it in the application.Sample method to make API call.

Future<http.Response> fetchPost() {

return http.get('https://jsonplaceholder.typicode.com/posts/1');

}

The response we get will be in the JSON format.All the data we receive from the request are stored in the http.response object.

Now lets create a function that makes a request to the server.

Future<Post> apiCall() async {

final response =

await http.get('https://jsonplaceholder.typicode.com/posts/1'); if (response.statusCode == 200) {

return Post.fromJson(json.decode(response.body));

} else {

throw Exception('Failed to load');

}

}

Calling the above function makes the api call to the server and converts the JSON Map into a Post using the fromJson() factory method only when the server returns “OK” with status code 200. If the server returns any unexpected response, error is thrown.

Parse the response :

Lets parse the response by converting the json response to dart objects. Create a model class to store the values from the response.

class Post {

final int userId;

final int id;

final String title;

final String body;



Post({this.userId, this.id, this.title, this.body});



factory Post.fromJson(Map<String, dynamic> json) {

return Post(

userId: json['userId'],

id: json['id'],

title: json['title'],

body: json['body'],

);

}

}

Now that we converted the response into dart objects, it can be used anywhere on the application.You can also see that it is very easy to fetch data from the internet using api.

If you find this post useful,Do hit the 👏 button and Follow to get more tutorials on Flutter.