Load CSV Data Into Elasticsearch
By: Sagar Gadewar
Tags: Bulk Import, Csv, Software Development, Elasticsearch, DevOps

Load CSV Data Into Elasticsearch

Import CSV Into AWS Elasticsearch Service Using Bulk import or Logstatsh Sagar Gadewar Follow Jul 5 · 4 min read

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.

Amazon Elasticsearch Service is a fully managed service that makes it easy for you to deploy, secure, and run Elasticsearch cost-effectively at scale.

Key Concepts

The key concepts of Elasticsearch are as follows −

Node: It refers to a single running instance of Elasticsearch. A single physical and virtual server accommodates multiple nodes depending upon the capabilities of their physical resources like RAM, storage, and processing power.

Cluster: It is a collection of one or more nodes. A cluster provides collective indexing and search capabilities across all the nodes for the entire data.

Index: It is a collection of different types of documents and their properties. The index also uses the concept of shards to improve performance. For example, a set of documents contains data of a social networking application.

Document: It is a collection of fields in a specific manner defined in JSON format. Every document belongs to a type and resides inside an index. Every document is associated with a unique identifier called the UID.

Shard: Indexes are horizontally subdivided into shards. This means each shard contains all the properties of a document but contains fewer JSON objects than an index. The horizontal separation makes the shard an independent node, which can be store in any node. The primary shard is the original horizontal part of an index and then these primary shards are replicated into replica shards.

Replicas: Elasticsearch allows a user to create replicas of their indexes and shards. Replication not only helps in increasing the availability of data in case of failure but also improves the performance of searching by carrying out a parallel search operation in these replicas.

There are multiple ways to load CSV data into Elasticsearch Cluster. I want to discuss two cases depends on the size of a file

If file size is of few KB’s (kilobytes) If file size in MBs or in GB

Prerequisites:

a) Create Elasticsearch

There are only a few basic steps to getting an Amazon Elasticsearch Service domain up and running:

Define your domain => Configure your cluster => Set up access => Review

refer: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html

You can also install Elasticsearch locally or host it on any virtual machine

Let's assume I have an Ubuntu machine locally

Case I

CSV file size < few KB’s

Elasticsearch expects some specific format of JSON file as input. To convert the CSV file into the required format we need to run the below script.

Install python3, Run below python script & Run curl command

Note: Provide file path of input_csv_path & input_json_path.

csv_to_es_json.py import csv

import json

import collections

orderedDict = collections.OrderedDict()

from collections import OrderedDict def csv_to_json(csvFilePath, jsonFilePath):

jsonArray = []

x = OrderedDict([('index', {})])

jsonString = json.dumps(x)

with open(csvFilePath, encoding='utf-8') as csvf:

with open(jsonFilePath, 'w', encoding='utf-8') as jsonf:

csvReader = csv.DictReader(csvf)

for row in csvReader:

jsonf.write(jsonString)

jsonf.write("

")

y = json.dumps(row)

jsonf.write(y)

jsonf.write("

")



csvFilePath = r'<input_csv_path>'

jsonFilePath = r'<input_json_path>'

csv_to_json(csvFilePath, jsonFilePath)

Output:

## input.csv

Title1,Title2,Title3

one,two,three

example1,example2,example3 ## output json

{ "index": {} }

{"Title1": "one", "Title2": "two", "Title3": "three"}

{ "index": {} }

{"Title1": "example1", "Title2": "example2", "Title3": "example3"}

Note: Provide ELASTIC_PASSWORD, DOMAIN_URL, INDEX_NAME DOC_TYPE, input_json_path these variables values in following command and run.

$ curl -u admin:<ELASTIC_PASSWORD> -H ‘Content-Type: application/json’ -XPOST ‘<DOMAIN_URL>/<INDEX_NAME>/<DOC_TYPE>/_bulk?pretty’ — data-binary @<input_json_path>`

eg.

curl -u admin:my_elastic_pass -H ‘Content-Type: application/json’ -XPOST ‘https://my-search-hieswgdbdgi8907fazkabdjknh.ap-south-1.es.amazonaws.com/myindex/_doc/_bulk?pretty' — data-binary @/path/output.json

Case II

CSV file size in MBs or GB

Create one VM or EC2 (at least 2 vCPU 4 GB RAM) of ubuntu 18..04

Install Logstash

sudo apt update

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install logstash

sudo apt-get install default-jdk

Create configuration file at location /etc/logstash/conf.d/import_csv.conf

input {

file {

path => "input_json_path"

start_position => beginning

}

}

filter {

csv {

columns => ["Title1","Title2","Title3"]

separator => ","

}

}

output {

elasticsearch {

hosts => ["DOMAIN_URL:443"]

index => "INDEX_NAME"

document_type => "DOC_TYPE"

ilm_enabled => false

ssl => true

user => "admin"

password => "ELASTIC_PASSWORD"

}

}

Restart Logstash service