markhneedham/blog huggingface dataset:
Apache Superset: Refusing to start due to insecure SECRET_KEY
Date: 2023-10-13 00:44:37
In this post, we'll learn how to configure a secret for using Apache Superset.

:icons: font

I've been trying to install https://pypi.org/project/apache-superset/[Apache Superset^] so that I can use it for a demo and ran into an issue with a secret key when trying to install it.
In this blog post, I'll explain how to work around it.

We're going to be using Poetry and will follow the https://superset.apache.org/docs/installation/installing-superset-from-scratch[installing from scratch^] guide.

First up is installing the library:

[source, bash]
----
poetry add apache-superset 
----

And then after initialising the database with `poetry run superset db upgrade`, we'll try to create the admin user:

[source, bash]
----
export FLASK_APP=superset                
poetry run superset fab create-admin
----

Running this command resulted in the following output:

.Output
[source, text]
----
--------------------------------------------------------------------------------
                                    WARNING
--------------------------------------------------------------------------------
A Default SECRET_KEY was detected, please use superset_config.py to override it.
Use a strong complex alphanumeric string and use a tool to help you generate 
a sufficiently random sequence, ex: openssl rand -base64 42
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Refusing to start due to insecure SECRET_KEY
----

I learnt that it wants me to specify the `SUPERSET_SECRET_KEY`, a problem that's been https://github.com/apache/superset/discussions/23598[encountered by several others^].
So let's set that environment variable and try again:

[source, bash]
----
export SUPERSET_SECRET_KEY="oh-so-secret" # <.>
poetry run superset fab create-admin
----
<.> I should probably use a more secret, but this is just for fun so I'll be lazy for now.

And this time it's happy:

.Output
[source, text]
----
logging was configured successfully
2023-10-13 13:54:52,555:INFO:superset.utils.logging_configurator:logging was configured successfully
2023-10-13 13:54:52,557:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/Users/markhneedham/Library/Caches/pypoetry/virtualenvs/data-app-er_2mvJz-py3.11/lib/python3.11/site-packages/flask_limiter/extension.py:336: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
Username [admin]: 
User first name [admin]: 
User last name [user]: 
Email [admin@fab.org]: 
Password: 
Repeat for confirmation: 
Recognized Database Authentications.
Admin User admin created.
----
Python/Poetry: Library not loaded: no such file, not in dyld cache
Date: 2023-07-27 11:44:37
In this post, we'll learn how to work around an error when using Python's Poetry library.

As I https://www.markhneedham.com/blog/2023/07/24/vscode-poetry-python-interpreter/[mentioned in a previous blog post^], I've been using Python's Poetry library, but today it stopped working!
In this blog post, I'll explain what happened and how I got it working again.

It started off innocent enough, with me trying to create a new project:

[source, bash]
----
poetry init
----

But instead of seeing the usual interactive wizard, I got the following error:

.Output
[source, text]
----
dyld[20269]: Library not loaded: /opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Python
  Referenced from: <1B2377F9-2187-39A9-AA98-20E438024DE2> /Users/markhneedham/Library/Application Support/pypoetry/venv/bin/python
  Reason: tried: '/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/Library/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/System/Library/Frameworks/Python.framework/Versions/3.11/Python' (no such file, not in dyld cache)
----

It seems to be looking for Python version 3.11.3 in a variety of different places.
I installed Python using brew, so let's check what's in that directory:

[source, bash]
----
ls -lh /opt/homebrew/Cellar/python@3.11
----

.Output
[source, text]
----
total 0
drwxr-xr-x@ 13 markhneedham  admin   416B 30 Jun 12:50 3.11.4_1
----

There isn't a Python 3.11.3 and I don't recall moving it, but it is possible that a `brew update` might have updated the Python version.
It also mentions that the Python version was referenced from `/Users/markhneedham/Library/Application Support/pypoetry/venv/bin/python`, which I thought might be a symbolic link that I could update.
Unfortunately, it's a binary file so we can't do that.

[source, bash]
----
file "/Users/markhneedham/Library/Application Support/pypoetry/venv/bin/python"
----

.Output
[source, text]
----
/Users/markhneedham/Library/Application Support/pypoetry/venv/bin/python: Mach-O 64-bit executable arm64
----


Luckily https://stackoverflow.com/questions/70920378/poetry-returns-dyld-library-not-loaded-image-not-found-following-brew-ins[StackOverflow came to the rescue^] with some classic turn it off and on again advice.
So let's first uninstall Poetry:

[source, bash]
----
curl -sSL https://install.python-poetry.org | python - --uninstall
----

And then reinstall it:

[source, bash]
----
curl -sSL https://install.python-poetry.org | python -
----

It works for me after doing this and Poetry's `python` executable now uses brew's version of Python:

[source, bash]
----
"/Users/markhneedham/Library/Application Support/pypoetry/venv/bin/python" --version
----

.Output
[source, text]
----
Python 3.11.4
----
Python: Re-import module
Date: 2023-07-07 04:44:37
In this post, we're going to learn how to re-import a local Python module

:icons: font

I often write little Python scripts that import code from other local modules and a common problem I have when using the Python REPL is that I update the code in the other modules and then can't use the new functionality without restarting the REPL and re-importing everything.
At least so I thought!
It turns out there is a way to refresh those modules and that's what we'll be exploring in this blog post.

Let's say that we have the file `person.py` that contains the code shown below:

.person.py
[source, python]
----
class Person:
  def __init__(self, name):
    self.name = name

  def say_hello(self):
    print(f"Hello {self.name}")
----

Launch the Python REPL by typing `python` and then import the class and run the `say_hello` function:

[source, python]
----
from person import Person

Person(name="Mark").say_hello()
----

.Output
[source, text]
----
Hello Mark
----

Now let's say that we add a `say_goodbye` function to the class so that it now looks like this:

.person.py
[source, python]
----
class Person:
  def __init__(self, name):
    self.name = name

  def say_hello(self):
    print(f"Hello {self.name}")

  def say_goodbye(self):
    print(f"Good Bye {self.name}")
----

If we try to use that function, it's not available:

[source, python]
----
from person import Person

Person(name="Mark").say_goodbye()
----

.Output
[source, text]
----
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Person' object has no attribute 'say_goodbye'
----

Even running `from person import Person` again doesn't help.
Help does come, however, from the https://docs.python.org/3/library/importlib.html[importlib^] library, which we can use to re-import the module.
I initially tried doing this:

[source, python]
----
import importlib

importlib.reload(person)
----

And then I ran the `say_goodbye` function again, to no avail:

.Output
[source, text]
----
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'person' is not defined. Did you mean: 'Person'?
----

To solve that problem, I had to first import `person` and then reload:

[source, python]
----
import person
importlib.reload(person)
----

I ran the goodbye command again, but still got the same error:

[source, python]
----
Person(name="Mark").say_goodbye()
----

.Output
[source, text]
----
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Person' object has no attribute 'say_goodbye'
----

It turns out that we need to re-run the import of the `Person` class again, after the reload of `person`, meaning that the full re-importing code looks like this:

[source, python]
----
import person
importlib.reload(person)
from person import Person
----

And now let's call goodbye:

[source, python]
----
Person(name="Mark").say_goodbye()
----

.Output
[source, text]
----
Good Bye Mark
----
VSCode: Adding Poetry Python Interpreter
Date: 2023-07-24 04:44:37
In this post, we're going to learn how to configure Poetry as a Python interpreter in VSCode.

:icons: font

I've been trying out Python's https://python-poetry.org/[Poetry^] dependency management tool recently and I really like it, but couldn't figure out how to get it setup as VSCode's Python interpreter.
In this blog post, we'll learn how to do that.

One way to add the Python interpreter in VSCode is to press `Cmd+Shift+p` and then type `Python Interpreter`. 
If you select the first result, you'll see something like the following:

.The Python interpreter modal window in VSCode
image::{{<siteurl>}}/uploads/2023/07/python-interpreter.png[]

When I create a virtual environment directly it'll usually appear on the list, but Poetry wasn't.
I went a-searching and came across https://stackoverflow.com/questions/59882884/vscode-doesnt-show-poetry-virtualenvs-in-select-interpreter-option[a StackOverflow thread where people were experiencing the same problem^].

I tried some of the suggestions, but they mostly didn't work for me.
I did, however, realise that I'd be able to add the interpreter manually if I knew its path, which led me to https://github.com/python-poetry/poetry/pull/731[this PR^].

This led me to the following command which gives us that answer:

[source, bash]
----
poetry env info 
----

.Output
[source, text]
----

Virtualenv
Python:         3.11.4
Implementation: CPython
Path:           /Users/markhneedham/projects/duckdb-0.8/sentiment-analysis/.venv
Executable:     /Users/markhneedham/projects/duckdb-0.8/sentiment-analysis/.venv/bin/python
Valid:          True

System
Platform:   darwin
OS:         posix
Python:     3.11.4
Path:       /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
Executable: /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/bin/python3.11
----

If we want to only get the path, we can pass in `--path`:

[source, bash]
----
poetry env info --path
----

.Output
[source, text]
----
/Users/markhneedham/projects/duckdb-0.8/sentiment-analysis/.venv
----

We can then pipe that to our clipboard:


[source, bash]
----
poetry env info --path | pbcopy
----

And now we can go back to the Python interpreter window and paste in the result.

Job done!
Puppeteer: Unsupported command-line flag: --enabled-blink-features=IdleDetection.
Date: 2023-07-13 04:44:37
In this post, we're going to learn how to work around a Chrome browser warning when using Puppeteer.

:icons: font


In many of the https://www.youtube.com/watch?v=6Ck1d5iswm4&list=PLihIrF0tCXdc35Lq865Z0jsnwaMol9Zn7[StarTree recipe videos^] that I've worked on, I show how to write queries in the Pinot UI.
If I wrote these queries manually there'd be way too many typos, so I drive the UI using a script.
I've recently been exploring whether I can do this using a Node.js library called https://pptr.dev/[Puppeteer^] and wanted to share a warning message that I ran into early doors.

I installed Puppeteer using the following command:

[source, bash]
----
npm i puppeteer-core
----

I then created the file `drive_pinot.mjs` and added the following code, which opens the Pinot query console on the first tab:

.drive_pinot.mjs
[source, javascript]
----
import puppeteer from 'puppeteer-core';

async function run() {
    const browser = await puppeteer.launch({
        headless: false,
        executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
        ignoreDefaultArgs: ['--enable-automation'], 
        defaultViewport: null, 
        args: [
            '--window-size=1920,1080',
            '--disable-infobars',
        ],
    });

    const [page] = await browser.pages(); // <1> 
    await page.goto('http://localhost:9000/#/query'); // <2>

    await page.waitForSelector('.CodeMirror');
    // await browser.close();
}

run();
----
<.> Get the first tab
<.> Open the Pinot query console

We can run that script like this:

[source, bash]
----
node drive_pinot.mjs
----

The script launched fine, but the Chrome browser had the following warning underneath the URL bar:

[source, text]
----
You are using an unsupported command-line flag: --enabled-blink-features=IdleDetection. Stability and security will suffer.
----

I didn't set any blink features, so I have no idea where this error came from.
By trial and error, I did work out that I could get rid of it by opening the Pinot query console on a new tab and then closing the initial tab.
Our code, therefore, looks like this:

.drive_pinot.mjs
[source, javascript]
----
import puppeteer from 'puppeteer-core';

async function run() {
    const browser = await puppeteer.launch({
        headless: false,
        executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',  // Add your path here
        ignoreDefaultArgs: ['--enable-automation'], // exclude this switch
        defaultViewport: null, // required for --window-size
        args: [
            '--window-size=1920,1080', // set the window size
            '--disable-infobars',
        ],
    });

    const page = await browser.newPage(); // <1> 
    await page.goto('http://localhost:9000/#/query');

    const [initPage] = await browser.pages();
    initPage.close() // <2> 

    await page.waitForSelector('.CodeMirror');
    // await browser.close();
}

run();
----
<.> Open a new browser tab
<.> Close the initial one

And now the error message has miraculously gone!
JupyterLab 4.0.5: Scroll output with keyboard shortcut
Date: 2023-09-03 00:44:37
In this post, we'll learn how to make the output of a Jupyterlab cell scrollable with a keyboard shortcut.

:icons: font

In the latest version of Jupyter Notebook/Lab (at least), the output of each cell is shown in full, regardless of how long it is.
I wanted to limit the height of the output and then scroll through it within that inner window, ideally by triggering a keyboard shortcut.

I learnt how to do this with the help of https://stackoverflow.com/questions/59025144/how-do-i-set-a-shortcut-to-enable-scrolling-for-outputs-in-jupyter-lab[Stack Overflow].
First, you need to open the settings editor by typing `Cmd + ,` on a Mac or by clicking on that screen from the top menu:

.Settings > Settings Editor
image::{{<siteurl>}}/uploads/2023/09/settings-editor.png[]

Next, click on `JSON Settings Editor` in the top right-hand corner:

.JSON Settings Editor
image::{{<siteurl>}}/uploads/2023/09/json-settings-editor.png[]

And then paste the following JSON into the `shortcuts` array:

[source, json]
----
        {
            "command": "notebook:enable-output-scrolling",
            "keys": [
                "S"
            ],
            "selector": ".jp-Notebook:focus",
            "args": {}
        },
        {
            "command": "notebook:disable-output-scrolling",
            "keys": [
                "Alt S"
            ],
            "selector": ".jp-Notebook:focus",
            "args": {}
        },
----

This is how it should look like once you've done that:

.Keyboard shortcuts added
image::{{<siteurl>}}/uploads/2023/09/json-edited.png[]

You can then add a scrollbar to an output cell by pressing `s` or remove the scrollbar by pressing `Alt + s`.
Below is an example of what happens when you toggle output scrolling:


.No scrollbar vs Scrollbar 
image::{{<siteurl>}}/uploads/2023/09/scrollbar-noscrollbar.png[]
How to run a Kotlin script
Date: 2023-09-07 00:44:37
In this post, we'll learn how to run a Kotlin script.

:icons: font

I was recently helping Tim get a https://github.com/tlberglund/pinot-movie-ratings[Pinot data-loading Kotlin script^] working and it took me a while to figure out the best way to run it.
In this blog post, I'll share the solution we came up with.

But first things first, we need to install Kotlin if it's not already installed.
I used a library called https://sdkman.io/[SDKMAN6] for all things JVM, so I'm gonna run the following command:

[source, bash]
----
sdk install kotlin
----

And now for the Kotlin script.
Our script prints out a JSON document representing a person:


.script.kts
[source, kotlin]
----
@file:DependsOn("com.google.code.gson:gson:2.8.6") // <.>

import com.google.gson.Gson

data class Person(val name: String)
class Blog {
    companion object {
        @JvmStatic
        fun main(args: Array<String>) {
            println(
                Gson().toJson(Person("Mark Needham"))
            )
        }
    }
}

println("Script started")
println("Before main function call")
Blog.main(arrayOf()) // <.>
println("After main function call")
----
<.> Install GSON as a dependency
<.> Run the `Blog` main class

After trying a few different approaches, I came https://github.com/kscripting/kscript[kscript^], a wrapper around `kotlinc` that takes care of compilation and dependency management for us.
We can install it like this:

[source, bash]
----
sdk install kscript
----

And then run our script:

[source, bash]
----
kscript script.kts
----

.Output
[source, text]
----
[kscript] Resolving com.google.code.gson:gson:2.8.6...
Script started
Before main function call
{"name":"Mark Needham"}
After main function call
----

And that's it!
kcat: SASL - Java JAAS configuration is not supported
Date: 2023-09-12 00:44:37
In this post, we'll learn how to ingest data into Kafka using kcat with SASL.

:icons: font

I've been updating the StarTree Kafka SASL recipe to use Pinot 0.12 and ran into an error while trying to have it use `kcat` to ingest data into Kafka.
In this blog post, we'll learn how I did this.

The initial recipe was ingesting data into Kafka using `kafka-console-consumer.sh`, which uses the Java Kafka client.
I'm using this Kafka client config file:

.kafka-config/kafka_client.conf
[source, properties]
----
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
        username="alice" \
        password="alice-secret";
----

And, we use this script to ingest data from a data generator:

[source, bash]
----
python datagen.py |
docker exec -i kafka-sasl /opt/kafka/bin/kafka-console-consumer.sh \
   --bootstrap-server localhost:9093  \
   --consumer.config /etc/kafka/kafka_client.conf \
   --topic events \
   --from-beginning
----

This all works fine, so I tried to update the script to use `kcat` instead:

[source, bash]
----
python datagen.py |
kcat -P -b localhost:9092 -F kafka-config/kafka_client.conf -t events
----

But this time it's not happy and we see the following error:

.Output
[source, text]
----
% Reading configuration from file kafka-config/kafka_client.conf
% ERROR: kafka-config/kafka_client.conf:3: Java JAAS configuration is not supported, see https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka for more information.
----

ChatGPT helped me translate my original configuration to the following:

.kafka-config/kafka_client_kcat.conf
[source, properties]
----
security.protocol=SASL_PLAINTEXT
sasl.mechanisms=PLAIN
sasl.username=alice
sasl.password=alice-secret
----

And then I updated my script to use that file:

```bash
python datagen.py |
kcat -P -b localhost:9092 -F kafka-config/kafka_client_kcat.conf -t events
```

If we run that for a few seconds, we can then check if any data has been ingested:

[source, text]
----
kcat -C -b localhost:9092 \
  -F kafka-config/kafka_client_kcat.conf \
  -t events \
  -c 5
----

.Output
[source, json]
----
{"ts": 1694529896154, "uuid": "1c192aca-05e4-497c-823c-6de697b38ebc", "count": 153}
{"ts": 1694529896154, "uuid": "1336fc02-6e2a-47da-bbdb-b960356f6cac", "count": 674}
{"ts": 1694529896154, "uuid": "ce4878f0-129a-4aad-b65f-d06b5817c5cd", "count": 231}
{"ts": 1694529896154, "uuid": "006d29dd-e6a2-4dec-b9dd-49ee72406aeb", "count": 287}
{"ts": 1694529896154, "uuid": "86693165-5194-41a5-9531-c61453b0d565", "count": 931}
----

Success!
JupyterLab 4.0.5: Adding execution time to cell
Date: 2023-08-20 00:44:37
In this post, we'll learn how to add execution time to cells in JupyterLab.

:icons: font

I've been using Jupyter Lab notebooks in some of my recent videos on https://www.youtube.com/@learndatawithmark/videos[Learn Data with Mark^] and I wanted to show cell execution timings so that viewers would have an idea of how long things were taking.
I thought I'd need to use a custom timer, but it turns out there's quite a nice plug-in, which we'll learn about in this blog post.

The plug-in is called https://github.com/deshaw/jupyterlab-execute-time[`jupyterlab-execute-time`^] and it shows a live view of the time that a cell takes to execute, as well as showing the execution time afterward.

.The jupyterlab-execute-time plugin-in in action
image::{{<siteurl>}}/uploads/2023/08/cell-execution-time.gif[]

We can install the library directly using pip:

[source, bash]
----
pip install jupyterlab_execute_time
----

Or Poetry, if that's your thing:

[source, bash]
----
poetry add jupyterlab_execute_time
----

Alternatively, we can use the extension manager inside Jupyter Lab, which you can launch by typing `Cmd + Shift + X` on a Mac.
Once it pops up, search for `jupyterlab-execute-time`, as shown below:

.Searching for the jupyterlab-execute-time plug-in
image::{{<siteurl>}}/uploads/2023/08/time-extension-manual-search.png[]

And then click on 'Install'.

Regardless of the installation option used, you should then see the plugin under the 'Installed' section:

.jupyterlab-execute-time plug-in installed
image::{{<siteurl>}}/uploads/2023/08/time-extension-installed.png[]

And here's a screenshot showing the timings for a few cells in a notebook that I'm currently working on:

.Multiple cell timings
image::{{<siteurl>}}/uploads/2023/08/jupyterlab-time-execution-cell.png[]