Tech Stack Behind HargaJoprak.com (…and Why It Is Used).
By: Ardhi
Tags: Startup, Lean Startup, JavaScript, MVP, Python

Tech Stack Behind HargaJoprak.com (…and Why It Is Used).

Homepage as this is written.

The What?

The MVP is built around the idea of price comparison site between marketplaces in Indonesia. Right now it’s only encompassing our four giants: Bukalapak, Tokopedia, Shopee, and Blibli, but soon more will be added in the future (if all is good).

It is the first MVP I released to the market to see the reception, validation, and personally, to asses the depth you can get in the fullstack development world + devops. Usually MVP is built within weeks, but for mine it took almost a full year and still hasn’t got it finished feel because of many parts still incomplete or ridden with bugs (well.. that could be part of learning).

A little more disclaimer, though. It’s pretty much still on beta phase (even pre-beta) with bugs and results are not 100% reliable (you can see below in ‘work in progress’), perhaps you’ll encounter lagging or even empty result or even unreachable site, but all of the component and intended concept is in there.

All in all, it’s created mainly for educational of purposes before jumping in to bigger projects.

The Reason

There aren’t so many marketplace comparison site in Indonesia and most of the major players if you go for “price comparison sites Indonesia”, mostly offering complex UI and I want to serve “right in your face” value without any fluff to the visitor, and so HargaJoprak was born. The aimed keywords for the project are reactive and informative, they’re something to be planted in mind every time the code is written.

search result for ‘samsung s10’ for Tokopedia

The Plan

Dozen or so languages and frameworks are tried and tested in this phase from Node, Go, Python, and frameworks like Aiohttp, Scrapy, Splash, before sticking out with old Python’s requests (with some twist) and Sanic asynchronous streaming response, because the keyword mentioned earlier demand for reactivity, so far Sanic delivers asynchronous requests with the ease of use of its API usage in the likes of Flask.

The Problem

Getting the data to scrape is the major pain in the arse at the beginning. At first I use Scrapy and admittedly it’s already mature and battle-tested tool, but unfortunately also added unnecessary complexities to this project, up to the point I was experimenting with saving the data to mongoDB via its middleware and re-collect it with document nested query to use a javascript renderer like BeautifulSoup and Splash to lazy-load Tokopedia results.

Later I found that was not needed. With proper planning this can be avoided.

Another thing to consider is the frontend management, it‘s like 70–80% of the headache, there are so many things to watch, states, CSS, SEO, UI/UX fluidity and reactivity, loading mechanism, async, etc 😅. I can write several posts on this topic alone, maybe we can just leave it as brief for now.

Search engine optimisation and getting to the point where the investment can be justified (paid off) is another problem. This yet to be foresee. I want to expand it further until it reach its optimum visitors for its size, how far it can get? (and how much juice can be squeezed?).

Google and search engines are notoriously famous for their fiendish behaviour towards single page applications (SPA). Well, I tend to think their bots should adapt to this instead of pre-rendered SPA solution, but until then Next.js is there to pre-render Javascript so crawling bots can read. Implementation is a little tricky because of the nature of dynamic page creation from the results (would you create rendered page each time the page is routed to display the results? Can it still be optimised with caching?)

Backend-wise, I need to uniformize the results it gets from each marketplace, because it’s not standardised, you can get various number of results, different fields (like no store owner field on BliBli), or different format (like rating or review is perceived differently between marketplaces).

Legality is also a concern, so I only scrape what’s allowed to be scraped which is in the robots.txt of each marketplace (you can see it in here) — just like what Google do with its crawling bots, but even with this, it’s still pretty much grey area and leaving homework to use other’s resources as less as possible.

The Development

React is used with Next.js to solve SEO problem and also React is fun to tinker about by juggling its props and states (I’ve tried Vue, but I feel like the style not really for me and couldn’t get along with its Nuxt.js— So far, IMO). In the future I should create room for Redux state management, as right now it’s enough to go on by a simple top-to-bottom component data flow approach.

I’m implementing de-coupling architecture to separate between frontend and backend as much as possible with only communicate via asynchronous streaming API calls (via newline delimited json) and just make room for future improvement like caching, because I want the data to be displayed as soon it’s retrieved from the marketplace to live the “reactivity” keyword.

Rough diagram.

Asynchronous nature is needed because you don’t get the same response from marketplaces. So, give the visitor the result as soon as available.

Hosted in Amazon EC2 on the nearest Singaporean hub so I can keep the hop short for mainly Indonesian visitors. AWS is easy also to scale-up and backup (right now just create bigger instance and re-attach volume) and for effective CI/CD is still on research and progress. Route53 is used to host the DNS zone with instant result, helped by AWS’ elastic IP. Nginx is used as a reverse proxy hardened with SSL.

Work in Progress

Q̵u̵e̵r̵y̵ ̵w̵i̵t̵h̵ ̵B̵l̵i̵B̵l̵i̵ ̵i̵s̵ ̵n̵o̵t̵ ̵s̵h̵o̵w̵n̵.̵

Uniformize the number of results and fields.

Create sitemap on-the-fly for SEO purpose.

Sort by price and best results from the server.

Test asynchronous results for multiple marketplaces.

Implement proper logging for error detection from visitors.

The Result So Far

Google analytics is used to track visitors. Promotion is kept minimum yet to catch what kind of error could possibly occur, so this part is still pretty much under development.

Thank you for reading.😊

UPDATE December 27th 2019:

The original concept largely has been abandoned because of the inefficiency of retrieving from each site.

I might implement crawling bot for each marketplace and feed to a database/log agregator separately but right now I’m doing it passively and moved to blogspot for its hosting.

Yes, the scraped content is now converted to blogspot .xml format you can import in the settings -> import content of your blogspot console page.

The advantage of this method is you don’t have to worry about hosting setup and management, even SSL certification already handled by Google, you only need to provide custom domain.

new looks of hargajoprak.com

I’ll reveal how it laid out on another post, in the meantime I’m pretty satisfied with the result and currently tweaking with its SEO to attract organic traffic.