markhneedham/blog huggingface dataset:
R: Numeric representation of date time
Date: 2014-12-13 19:58:13

I've been playing around with date times in R recently and I wanted to derive a numeric representation for a given value to make it easier to see the correlation between time and another variable.

e.g. December 13th 2014 17:30 should return 17.5 since it's 17.5 hours since midnight.

Using the standard R libraries we would write the following code:

[source,r]
----

> december13 = as.POSIXlt("2014-12-13 17:30:00")
> as.numeric(december13 - trunc(december13, "day"), units="hours")
[1] 17.5
----

That works pretty well but https://twitter.com/tonkouts[Antonios] recently introduced me to the http://cran.r-project.org/web/packages/lubridate/index.html[lubridate] so I thought I'd give that a try as well.

The first nice thing about lubridate is that we can use the date we created earlier and call the +++<cite>+++floor_date+++</cite>+++ function rather than +++<cite>+++truncate+++</cite>+++:

[source,r]
----

> (december13 - floor_date(december13, "day"))
Time difference of 17.5 hours
----

That gives us back a +++<cite>+++difftime+++</cite>+++\...

[source,r]
----

> class((december13 - floor_date(december13, "day")))
[1] "difftime"
----

\...which we can divide by different units to get the granularity we want:

[source,r]
----

> diff = (december13 - floor_date(december13, "day"))
> diff / dhours(1)
[1] 17.5

> diff / ddays(1)
[1] 0.7291667

> diff / dminutes(1)
[1] 1050
----

Pretty neat!

lubridate also has some nice functions for creating dates/date times. e.g.

</p>

[source,r]
----

> ymd_hms("2014-12-13 17:00:00")
[1] "2014-12-13 17:00:00 UTC"

> ymd_hm("2014-12-13 17:00")
[1] "2014-12-13 17:00:00 UTC"

> ymd_h("2014-12-13 17")
[1] "2014-12-13 17:00:00 UTC"

> ymd("2014-12-13")
[1] "2014-12-13 UTC"
----

And if you want a different time zone that's pretty easy too:

[source,r]
----

> with_tz(ymd("2014-12-13"), "GMT")
[1] "2014-12-13 GMT"
----
Dash: Configurable dcc.Interval
Date: 2022-04-23 02:44:37
In this post we'll learn how to make the timeout on Dash's dcc.Interval component configurable.

:icons: font

As I mentioned in my blog post about https://www.startree.ai/blogs/building-real-time-crypto-dashboard-pinot-dash[building a Real-Time Crypto Dashboard^], I've recently been working with the https://dash.plotly.com/[Dash^] low-code framework for building interactive data apps.

I was using the https://dash.plotly.com/dash-core-components/interval[`dcc.Interval`^] component to automatically refresh components on the page and wanted to make the refresh interval configurable.
In this blog post we'll learn how to do that.

.Dash: Configurable dcc.Interval
image::{{<siteurl>}}/uploads/2022/04/dash-banner.png[]


== Setup

Let's first setup our Python environment:

[source, bash]
----
python -m venv .env
source .env/bin/activate
----

And install the dash library:

[source, bash]
----
pip install dash
----

== Building a refreshing dashboard

Now we're going to build a simple dashboard that refreshes a timestamp on the page once every 1,000 milliseconds i.e. once per second.
The code to do this is shown below:

.dashboard.py
[source, python]
----
include::content/2022/04/23/dashboard.py[]
----

[source, bash]
----
python dashboard.py
----

If we navigate to http://localhost:8050, we'll see the timestamp updating once per second, as shown in the following animation:

.Refreshing every second
image::{{<siteurl>}}/uploads/2022/04/dash-once-second.gif[]

The timestamp updates once every second, just as we'd expect.

To make this value configurable we will use `dcc.Slider` to set the new refresh rate and we'll add a new callback that updates the refresh interval.

.dashboard.py
[source, python]
----
include::content/2022/04/23/dashboard2.py[]
----
<1> `dcc.Slider` lets us configure a new refresh rate
<2> This callback updates the value of `interval-component`

Now if we navigate back to our web browser, we can adjust the interval to be whatever we like:

.Refreshing at different intervals
image::{{<siteurl>}}/uploads/2022/04/adjustable-interval.gif[]
Streamlit: Overwrite previous value in a loop
Date: 2022-02-19 02:44:37
In this post we'll learn how to write out to the screen only the latest value in a loop of values using Streamlit.

I was recently building a Streamlit app in which I was looping through a stream of values and wanted to only print out the most recent value.
In this blog post we'll learn how to do that.

.Streamlit: Overwrite previous value in a loop
image::{{<siteurl>}}/uploads/2022/02/streamlit-banner.png[]

== Setup

If you want to play along you'll need to create a virtual environment and install Streamlit:

[source, bash]
----
python -m  venv env
source venv/bin/activate
pip install streamlit
----

== Streamlit App

Now, create a file `app.py` that contains the following:

.app.py
[source, python]
----
import streamlit as st

st.markdown("#### Numbers one after the other")
for i in range(0,10):
  st.write(i)
----

And run the following command from the terminal:

[source, bash]
----
streamlit run app.py
----

If we run this command it will open a web browser at http://localhost:8501, and we'll see the following output:

.Numbers one after the other
image::{{<siteurl>}}/uploads/2022/02/numbers-one-after-the-other.png[width="300px"]

The numbers are all printed out, which isn't exactly what we want.
In this example we're only printing out 10 numbers, but if we had a bigger stream of values it would eventually crash the browser!

Instead we can use a single element container (https://docs.streamlit.io/library/api-reference/layout/st.empty[`st.empty`]) and populate that on each loop.
Let's update `app.py` to look like this:

.app.py
[source, python]
----
import streamlit as st

left, right = st.columns(2)

with left:
  st.markdown("#### Numbers one after the other")
  for i in range(0,10):
    st.write(i)

with right:
  st.markdown("#### Numbers overwriting each other")
  numbers = st.empty()

  for i in range(0,10):
    with numbers.container():
      st.write(i)
----

Now if we go back to our Streamlit app, we'll see the following:

.Numbers overwriting each other
image::{{<siteurl>}}/uploads/2022/02/numbers-overwriting-each-other.png[width="600px"]

We can only see the number 9 because that's the last value that got printed.

If we want to see how the output progresses, we'll need to pause the loop for a little bit after each iteration.
Update `app.py` to look like this:

.app.py
[source, python]
----
import streamlit as st
import time

left, right = st.columns(2)

with left:
  st.markdown("#### Numbers one after the other")
  for i in range(0,10):
    st.write(i)

with right:
  st.markdown("#### Numbers overwriting each other")
  numbers = st.empty()

  for i in range(0,10):
    with numbers.container():
      st.write(i)
      time.sleep(0.5)
----

And now if we refresh the page, we'll see it update like this:

.Numbers overwriting each other animation
image::{{<siteurl>}}/uploads/2022/02/sleep-numbers.gif[width="600px"]

Exactly what I wanted - good work Streamlit!
Apache Pinot: Inserts from SQL - Unable to get tasks states map - No task is generated for table
Date: 2022-11-18 02:44:37
In this post we'll learn how to fix the Unable to get tasks states map - No task is generated for table error when using Apache Pinot's inserts from SQL feature.

I recently wrote a post on the StarTre blog describing the https://startree.ai/blog/apache-pinot-0-11-inserts-from-sql[inserts from SQL^] feature that was added in Apache Pinot 0.11, and while writing it I came across some interesting exceptions due to configuration mistakes I'd made.
In this post we're going to describe one of those exceptions.

To recap, I was trying to ingest a bunch of JSON files from an S3 bucket using the following SQL query:

[source, sql]
----
INSERT INTO "events"
FROM FILE 's3://marks-st-cloud-bucket/events/*.json'
OPTION(
  taskName=myTask-s3,
  input.fs.className=org.apache.pinot.plugin.filesystem.S3PinotFS,
  input.fs.prop.accessKey=AKIARCOCT6DWLUB7F77Z,
  input.fs.prop.secretKey=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,
  input.fs.prop.region=eu-west-2
);
----

[NOTE]
====
Don't worry, those credentials were deactivated and deleted several days ago.
====

When I ran this query against a Pinot cluster that contained a controller, broker, and server, I got the following exception:

[source, text]
----
[
  {
    "message": "QueryExecutionError:\norg.apache.commons.httpclient.HttpException: Unable to get tasks states map. Error code 400, Error message: {\"code\":400,\"error\":\"No task is generated for table: events, with task type: SegmentGenerationAndPushTask\"}\n\tat org.apache.pinot.common.minion.MinionClient.executeTask(MinionClient.java:123)\n\tat org.apache.pinot.core.query.executor.sql.SqlQueryExecutor.executeDMLStatement(SqlQueryExecutor.java:102)\n\tat org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:145)\n\tat org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:103)",
    "errorCode": 200
  }
]
----

My mistake here was that I **didn't have a minion** in the cluster.
The ingestion job is run by the minion component, so without one of those this feature doesn't work!

== An update (30th June 2023)

Today I learned that you can get this error even if you do have a minion configured.
The scenario that results in this error is if no files are found for ingestion. 

This might happen if you have an invalid glob expression in the `includeFileNamePattern`:
For example, the following throws the exception:

[source, sql]
----
SET taskName = 'events-task7';
SET input.fs.className = 'org.apache.pinot.spi.filesystem.LocalPinotFS';
SET includeFileNamePattern='glob:customers.csv';
INSERT INTO customers
FROM FILE 'file:///input/';
----

We can fix the query by adding `**/` at the beginning:

[source, sql]
----
SET taskName = 'events-task7';
SET input.fs.className = 'org.apache.pinot.spi.filesystem.LocalPinotFS';
SET includeFileNamePattern='glob:**/customers.csv';
INSERT INTO customers
FROM FILE 'file:///input/';
----
Java: FileSystems.getDefault().getPathMatcher: IllegalArgumentException
Date: 2022-11-11 02:44:37
In this post we'll learn how to use Java's FileSystems.getDefault().getPathMatcher

I was debugging something in the https://github.com/apache/pinot/[Apache Pinot code^] earlier this week and came across the `FileSystems.getDefault().getPathMatcher` function, which didn't work quite how I expected. 

The function creates a `PathMatcher` that you can use to match against Paths.
I was passing through a value of `*.json`, which was then resulting in code similar to this:

[source, java]
----
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.PathMatcher;

class Main {  
  public static void main(String args[]) { 
    PathMatcher matcher = FileSystems.getDefault().getPathMatcher("*.json");
    System.out.println(matcher.matches(Path.of("/tmp", "mark.json")));
  } 
}
----

I've copied that code into https://replit.com/@MarkNeedham/Mark-Needham#Main.java[Replit^] and if we run it on there, we'll see the following exception:

[source, text]
----
Exception in thread "main" java.lang.IllegalArgumentException
    at java.base/sun.nio.fs.UnixFileSystem.getPathMatcher(UnixFileSystem.java:286)
    at Main.main(Main.java:5)
exit status 1
----

After reading the https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html#getPathMatcher(java.lang.String)[documentation^] I learnt that the pattern needs to be prefixed with `regex:` or `glob:`, otherwise it won't work. 
Let's update the example to do that:

[source, java]
----
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.PathMatcher;

class Main {  
  public static void main(String args[]) { 
    PathMatcher matcher = FileSystems.getDefault().getPathMatcher("glob:*.json");
    System.out.println(matcher.matches(Path.of("/tmp", "mark.json")));
  } 
}
----

If we run that again, this time we get a result telling us that the file doesn't exist:

[source, text]
----
false
----
Docusaurus: Side menu on custom page
Date: 2022-07-11 02:44:37
In this post we'll learn how to add a side menu to a custom page with the Docusaurus static website generator.

I've been working with Docusaurus to build the https://dev.startree.ai/[dev.startree.ai] website over the last few months and I wanted to add https://docusaurus.io/docs/creating-pages[a custom page^] with a sidebar similar to the one that gets automatically generated on documentation pages. 

All the examples I could find showed you to create a splash page, so it took me a while to figure out how to do what I wanted, but in this post we'll learn how to do it.

.Docusaurus: Side menu on custom page
image::{{<siteurl>}}/uploads/2022/07/docusaurus-banner.png[]

First, let's create a Docusaurus site called `my-website` by running the following command:

[source, bash]
----
npx create-docusaurus@latest my-website classic
----

We can run the site locally with the following command:


[source, bash]
----
npm run start
----

This will run a local webserver at http://localhost:3000, which will show the following page:

.Docusaurus
image::{{<siteurl>}}/uploads/2022/07/docusaurus.png[]

Custom pages go under `src/pages` and whatever we name our file will be the name of the page. 
Let's create `myCustomPage.js` with the following contents:

.src/pages/myCustomPage.js
[source, js,indent=2]
----
include::content/2022/07/11/myCustomPage.js[]
----

If we want to adjust the items show in the side bar we'd need to change the `DocSidebar` component's `sidebar` property. 
Most of the code shown here describes CSS styles that do change location between minor versions.
I'll try to keep the example up to date with the latest version.

[NOTE]
====
The version of the docusaurus at the time of writing this blog post is 2.0.0-beta.22.
====

The custom page is created at http://localhost:3000/myCustomPage and looks like this:

.Docusaurus Custom Page
image::{{<siteurl>}}/uploads/2022/07/docusaurus-myCustomPage.png[]
Python: Generate WKT from Lat Long Coordinates
Date: 2022-01-14 00:44:37
In this post we'll learn how to convert lat long coordinates to the WKT format.

Recently I've been playing around with geometry objects in https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry[WKT format^] while documenting https://docs.pinot.apache.org/basics/indexing/geospatial-support[Apache Pinot's Geospatial functions^].
I then wanted to figure out how to generate a WKT string from a list of lat long coordinates, which we'll learn how to do in this blog post.  

.Python: Generate WKT from Lat Long Coordinates
image::{{<siteurl>}}/uploads/2022/01/wkt-banner.png[]

We're going to do all this using Python's https://shapely.readthedocs.io/en/stable/manual.html[Shapely^] library, so let's first install that library:

[source, bash]
----
pip install shapely
----

[NOTE]
====
I learnt about the Shapely library from https://gis.stackexchange.com/questions/294206/create-a-polygon-from-coordinates-in-geopandas-with-python[Davma's StackOverflow question^].
====

Let's say we have a bunch of lat/long coordinates defined like this:

[source, python]
----
lat_longs = [
    (50.854457, 4.377184), 
    (52.518172, 13.407759), 
    (50.072651, 14.435935), 
    (48.853033, 2.349553), 
    (50.854457, 4.377184)
]
----

We're going to create a polygon based on these coordinates using the following code:

[source, python]
----
from shapely.geometry import Polygon
polygon = Polygon(lat_longs)
----

And getting a WKT string is as easy as calling the `wkt` property:

[source, python]
----
print(polygon.wkt)
----

.Output
[source, text]
----
POLYGON ((50.854457 4.377184, 52.518172 13.407759, 50.072651 14.435935, 48.853033 2.349553, 50.854457 4.377184))
----

And finally, we can plot that polygon on a map using the https://clydedacruz.github.io/openstreetmap-wkt-playground/[OpenStreetMap WKT Playground^].

.Polygon visualisation
image::{{<siteurl>}}/uploads/2022/01/wkt-diagram.png[width="500px"]
docker exec: Passing in environment variables
Date: 2022-06-16 02:44:37
In this post we'll learn how to pass environment variables to docker exec.

I've been working on an https://dev.startree.ai/docs/pinot/recipes/[Apache Pinot recipe^] showing how to ingest data from S3 and I needed to pass in my AWS credentials to the `docker exec` command that I was running.
It wasn't difficult to do, but took me a little while to figure out.

.docker exec: Passing in environment variables
image::{{<siteurl>}}/uploads/2022/06/docker-exec.png[]

The command that I was running looked like this:

[source, bash]
----
docker exec \
  -it pinot-controller bin/pinot-admin.sh LaunchDataIngestionJob \
  -jobSpecFile /config/job-spec.yml
----

And the https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3[Pinot documentation^] says that I need to pass in `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
I initially came across https://stackoverflow.com/questions/27812548/how-to-set-an-environment-variable-in-a-running-docker-container[a StackOverflow question^] that described a bunch of approaches that didn't work for me, but luckily I found https://devconnected.com/docker-exec-command-with-examples/[a tutorial^] that showed how to do it. 

Effectively we can set these environment variables via the `-e` parameter, as shown below: 

[source, bash]
----
docker exec \
  -e AWS_ACCESS_KEY_ID=<accessKeyId> \
  -e AWS_SECRET_ACCESS_KEY=<secretAccessKey> \
  -it pinot-controller bin/pinot-admin.sh LaunchDataIngestionJob \
  -jobSpecFile /config/job-spec.yml
----

It's that simple!
kcat/jq: Reached end of topic at offset: exiting
Date: 2022-12-06 02:44:37
In this post we'll learn how to work around a problem when using kcat and jq together.

I've recently been working with Debezium to get the https://dev.startree.ai/docs/pinot/demo-apps/pizza-shop[Pizza Shop product catalogue^] from MySQL into Apache Kafka and ran into an issue when querying the resulting stream using https://docs.confluent.io/platform/current/app-development/kafkacat-usage.html[kcat^] and https://stedolan.github.io/jq/[jq^].
In this blog I'll show how I worked around that problem.

I configured Debezium to write any changes to the products table into the `mysql.pizzashop.products` topic. 
I then queriesthis topic to find the changes for just one of the products:

[source, bash]
----
kcat -C -b localhost:29092 -t mysql.pizzashop.products -e  | jq '.payload | select(.id == 1)'
----

And got the following output:

.Output
[source, bash]
----
{
  "id": 1,
  "name": "Moroccan Spice Pasta Pizza - Veg",
  "description": "A pizza loaded with a spicy combination of Harissa sauce and delicious pasta.",
  "category": "veg pizzas",
  "price": 335,
  "image": "https://www.dominos.co.in//files/items/MoroccanSpicePPVG_N.jpg",
  "created_at": "2022-11-22T10:07:36Z",
  "updated_at": 1669111656000,
  "__deleted": "false"
}
% Reached end of topic mysql.pizzashop.products [0] at offset 83: exiting
{
  "id": 1,
  "name": "Moroccan Spice Pasta Pizza",
  "description": "A pizza loaded with a spicy combination of Harissa sauce and delicious pasta.",
  "category": "veg pizzas",
  "price": 335,
  "image": "https://www.dominos.co.in//files/items/MoroccanSpicePPVG_N.jpg",
  "created_at": "2022-11-22T10:07:36Z",
  "updated_at": 1669117483000,
  "__deleted": "false"
}
----

If I took away the `jq` command I wasn't getting the 'reached end of topic' error message, it was returning both events properly.
But the messages don't format so nicely without jq so I still needed a solution.

A bit of searching led me to https://ftisiot.net/posts/jq-kcat-consumer/[a blog post^] written by Francesco Tisiot in which he described exactly the same problem that I was having.
The solution is to pass in the `-u` flag to kcat, which stops it from buffering the output.
We end up with the following command that works perfectly:

[source, bash]
----
kcat -C -b localhost:29092 -t mysql.pizzashop.products -e -u  | jq '.payload | select(.id == 1)'
----

Thanks Francesco!
nvim: Unable to create directory for swap file - recovery impossible: permission denied
Date: 2024-01-03 00:44:37
In this post, we'll learn how work about a permissions issue when using nvim.

:icons: font

I was playing around with https://neovim.io/[neovim^] last week and despite installing it via Homebrew, ran into a weird permissions error.
In this blog post, I'll describe the problem I had and how to solve it.

I installed it like this:

[source, bash]
----
brew install nvim
----

And then tried to create a new file:

[source, bash]
----
nvim foo.py
----

Which resulted in the following error:

.Output
[source, text]
----
E303: Unable to create directory "/Users/markhneedham/.local/state/nvim" for swap file, recovery impossible: permission denied
E303: Unable to open swap file for "foo.py", recovery impossible
----

Hmm, let's take a look at what's in my `~/.local` directory.

[source, bash]
----
ls -alh ~/.local
----

.Output
[source, text]
----
total 0
drwxr-xr-x    6 markhneedham  staff   192B 29 Nov 21:40 .
drwxr-x---+ 149 markhneedham  staff   4.7K  2 Jan 23:37 ..
drwxr-xr-x@   5 markhneedham  staff   160B 19 Dec 06:22 bin
drwxr-xr-x@   6 markhneedham  staff   192B 29 Nov 21:40 pipx
drwxr-xr-x    5 markhneedham  staff   160B  2 Jan 23:31 share
drwxr-xr-x    3 root          staff    96B 14 Feb  2023 state
----

Weird, it's made `root` the owner of the `state` directory, whereas I need it to be `markhneedham`.
But we can fix that using `chown`:

[source, bash]
----
sudo chown  ~/.local/state markhneedham:staff
----

And now if we open a file with nvim it's happy!