Lightning Chronicles 1 : A routing node operator’s adventures.
By: Guillaume Girard
Tags: Node, Lightning Network, Ln, Bitcoin, Routing

II/ Routin’Omics 101: Managing liquidity, choosing the right peers, and defining a sustainable fee policy. (Guillaume)

This part will walk you through the decisions we’ve made concerning 3 distinct parameters of a routing node: Channels, Fees, and Liquidity.

In the process of documenting the activity of our node, I’ve identified 3 different phases that match the progress we’ve made since we began this adventure.

The first one I decided to name: The Savage State.

Yes, the savage state was the very beginning of the KryptoSphere Node, we had pretty much no idea of what we were doing and just like savages, we paid the cost of not understanding the environment that surrounded us.

The first step of the process was to establish a list of potential nodes with who to connect. But how can you choose a list of viable candidates? Your goal is to route as much as possible, so you obviously want to choose a big, healthy, and well-connected node. Or do you ..? It goes without saying that all rational actors in the network will try to adopt a profit-maximizing strategy, but remember, our initial goal was not to make a profit/generate a form of passive income. So, we tried to identify “undervalued” nodes with the most routing potential that were not already flooded with liquidity.

For that, we used our knowledge of the industry and the few tools at our disposal. Having followed the evolution of LN since 2018, we had a pretty good idea of what to look for in terms of growing companies operating a Lightning node. (if not already you should subscribe to both LightningLabs and LNMarkets newsletters to never miss what’s happening on LN).

Given the anticipation for a new bull market and the probable subsequent rise in fees on L1, we decided to focus on exchanges, betting on the trend of LN integration that would drastically reduce fees for users. We figured nobody would accept sky-high fees for a $20 transaction like in 2017. We also looked for emerging companies on LN that could see a meaningful increase in the use/sale of their product in the coming months (in retrospect it was inevitable).

With the help of 1ML, the Acinq Explorer, and one of the brightest mind in Bitcoin a.k.a Romain Rouphael (LNMarkets Co-Founder) we settled on the following companies :

LNMarkets: It was a no-brainer really, betting on the growth of LNM we chose to allocate 4,231,948 sats.

Southxchange: We identified Southxchange on Twitter and looked at their stats on 1ML. We allocated 5,302,384 sats. They are really well connected, have a lot of liquidity, but most importantly we believe that their geographical location is very promising. (BTC is booming in Argentina)

Tippin.me: We all knew Tippin prior to building the node, we know it is well known in the community and has a good chance of seeing a lot of volume. 5,134,843 sats allocated

Fold: Don’t think I have to justify why we chose Fold. Even though it was an already well-connected node, we believed that their potential for exponential growth justified allocating 5,037,248 sats. Spin the wheel!

BCash_Is_Trash: The name by itself was enough to convince us. With 6,103,938 sats allocated, this node was very high on the BOS score list, well-connected, and very healthy overall.

WalletOfSatoshi.com [2]: Well respected LN Mobile wallet, underrated at the time in our opinion. Justifying a 5,024,814 sats allocation. [NOW CLOSED]

Bitstamp: They had just integrated LN earlier in the year, based on our hypothesis we thought this node will get a lot of volume. 2,194,284 sats. [NOW CLOSED]

OpenNode: As an underrated competitor to the already crowded BTCPay node, we believed OpenNode had a lot of potential by facilitating merchant adoption. 3,894,562 sats. [NOW CLOSED]

JohnOnChain: A fellow Frenchman that was already routing a lot and accepted to open a channel with us. 1,957,452 sats. [NOW CLOSED]

The River channel opening marked the beginning of the golden-era for our node. Much routing, sats go up. I’m a long-time admirer of River, I think they have a unique positioning and customer experience. They were one the first exchanges to see the potential and superiority of the LN for their clients, so, I tried to contact them directly on Twitter to convince them to open a channel with us and they did!

We also opened a 5 million sats channel with Zebedee. Zebedee is a Lightning payment processor, with a focus on Lightning-enabled games, where users can win real satoshis for example. Our decision to open a channel with them was led by the love of Fanis for Bitcoin and Lightning infused games, some of which leverage Zebedee’s solution to send payments to players. We also thought it would be a great peer to connect to, with a lot of traffic but still a bit “under the radar” (compared to big nodes such as Bitrefill for example).

Upon opening the channel, we found out that the fee on their side was quite big (1000 ppm). Suspecting that it was only because they might have some automatic policy for unsolicited channels, we contacted Zedebee and they (thanks to Big Boss André) quickly set a lower fee to 1ppm. Moral of the story, watch out for your peer’s fees!

Something to watch out for is the Channel Acceptor parameter: While we initially set up ours to 1M sats (something about the median size of channels I’m told) we later decided to lower it to 500k in case we missed out on volume generating channels.

Fee policy :

The first choice we’ve made when it comes to fees was to reduce both the base fee and variable fee to 0.001. We probably thought we were geniuses at the time, thinking we would route a lot more payments this way. What happened, as indicated by the name of the era, is that we got brutalized. People used our node to make huge payments and drained all of our liquidity in a matter of days.

Some valuable insight we derived from this was that if you want to route a lot of payments but not drain your liquidity, you can set your base fee to 0 and raise your variable fee so that it will disincentivize big payments. As we raised our variable fee to 100ppm, we witnessed a steep decline in routed volume, so we adjusted down to 20, and again after that to 5, which has been our fee level for quite some time now. We plan to raise this in the future but as of now, it seems to bear good results.

How to rebalance channels?

Example of a Circular Rebalancing from Zedebee to Fold.

Circular Rebalancing: Using Balance of Satoshi by Alex Bosworth we used this type of off-chain rebalancing very early on. Because you need to pay a fee for each “hop” it can rapidly become quite expensive to rebalance your channels this way. However, in a high on-chain fee environment like today, it might be cheaper to use circular rebalancing . In this case, we recommend that you check the different “bos routes” proposed and choose only the cheaper ones. (https://github.com/alexbosworth/balanceofsatoshis)

Using Balance of Satoshi by Alex Bosworth we used this type of off-chain rebalancing very early on. Because you need to pay a fee for each “hop” it can rapidly become quite expensive to rebalance your channels this way. However, in a high on-chain fee environment like today, . In this case, we recommend that you check the different “bos routes” proposed and choose only the cheaper ones. (https://github.com/alexbosworth/balanceofsatoshis) Loop: Developed by Lightning Labs (of course), Loop is a tool that allows you to rebalance your channels using the Bitcoin blockchain. More info here: https://lightning.engineering/loop/ . We mostly used Loop to rebalance our channels because at the time fees were quite low on the Bitcoin blockchain. By using the command line you can avoid using the “fast mode” enabled by default and save some fees. Most of our Loops cleared at 3 sats/Vbyte or cheaper.

NamsBreakEvenPoint:

In order to measure how effective you are at running a routing node, you can use the NamsBreakEvenPoint. For the first phase of our experiment, we decided to focus on volume rather than fees but we will definitely use it in the future. It is absolutely necessary to measure if the cost of rebalancing your channels is not higher than the fees you’ve collected. It would be shameful to disclose our current Nams ratio, so I will spare you this pain…

Then we kinda lost motivation, enter: The Winter blues Era.

Yeah…We did nothing for an entire month and left our beloved node to rot. As we witnessed the volume of sats routed plummet to zero after our last adjustment, we lost our mojo. As far as we were concerned, we failed and our node was “dead”. Or so we thought…

Turns out, we are idiots. The node was displaying no volume due to a scripting error and our strategy was actually proving to work very well! I’ll let Fanis explain to you what happened. (totally his fault btw, you can hold him accountable for the evident lack of content in this part. Should’ve known not to trust the Greeks with money since 2011…)

Read The Fucking Manual

To track the evolution of the volume of payments routed by our node, we designed a basic script that uses lnd fwdinghistory command to retrieve forwarded payments information and display it in a nice table. But the number of forwarded payments stopped to increase at exactly 100. Eventually, we figured out that the fwdinghistory command defaults to displaying only the first 100 forwarded payments at a time. Running the same command but specifying that we wanted to see the 50,000 last forwarded payments, we discovered that our node had actually been routing a lot of new payments (around 800 payments) in the past months. We just couldn’t see it.

There’s a quite simple, generalizable lesson here: when you use a tool, carefully read the documentation. The fact that the fwdinghistory command needs to be supplied with a max_events argument, and defaults to 100 events, was in plain sight all this time.

And so here we are. You know everything that happened until now.

Enter: The Rebirth Phase

Once we discovered the scripting error, we regained confidence in our ability to become once again routing overlords. Some house cleaning had to be done before we could continue normal operations (ie. rebalancing some channels and re-evaluate our current strategy). To evaluate our current strategy we started looking at the way we managed our liquidity because while we were away, we noticed that some channels tended to balance themselves without our intervention while others remained completely unbalanced after a single large payment.

Liquidity Management :

We learned very fast that maintaining a routing node was largely dependent on how you can manage your liquidity because, without liquidity, your channels are useless.

One thing that you could use to reduce the burden of liquidity management is the AutoRebalanceQualityRatio.

AutoRebalanceQualityRatio:

|x| = absolute value of x

We used this ratio to determine if a channel can rebalance itself in order to reduce rebalancing costs over time. The closer the ratio is to 0, the better. It could become a viable strategy for routing operators looking to save on rebalancing costs to only allocate funds to channels with a ratio close to 0.

For example: While LN Market is a performing channel in terms of routed volume, with a ratio of 0.30, some maintenance in the form of rebalancing costs might be needed to keep routing. On the other hand, a channel like Zedebee, comparable in terms of routed volume, has a much better ratio of 0.037 (!). Now, keep in mind that this is an average over time and there are other factors to consider when evaluating the efficiency of a channel (a channel with a ratio of 1, like River Financial, can still route a lot of payments, you’ll just have to rebalance it regularly).

Routing Analysis :

Something that we also started to monitor is the Girard Ratio (named shamelessly after yours truly). The Girard ratio is a measure of the overall routing efficiency of a node (or individual channel) by comparing how much Bitcoin you locked on LN to how much you’ve actually routed for this amount of BTC. The higher the number, the better. For example, a ratio of 3 would indicate that for 1000 sats allocated to a channel, this channel routed over 3000 sats. Much wow, routing go up!

Girard Ratio:

TotalLiquidityEngagedInChannels is different from the capacity because it is only the funds that *we* devoted to channels. It doesn’t take the funds of our peers into account, eg channels that were opened with us by another node aren’t counted.”

The current Girard Ratio is 5.97 indicating that our node has routed on average 6 times the amount of Bitcoin we allocate for it.

For the Girard Ratio of a channel in particular, things can be a bit different. The goal is to be able to compare two channels on their respective Girard ratios. We believe a peer that receives a lot of payment (eg. with a big “forwarded out” amount) should be treated as equally important as a peer that sends a lot of payment (eg. with a big “forwarded in” amount), simply because they are just the two sides of the same transaction that passes through our node. Therefore, it seems appropriate to take the sum of both “forwarded in” and “forwarded out” parts as the “TotalForwardedVolume”, and divide by the number of sats we locked into this channel.

You can also use the ratio to compare channels, for example, LNMarket has a ratio of 8.85 while Zebedee has a lower ratio of 3.02. This goes to show that while Zedebee has more auto-rebalancing capabilities, LNMarkets is just the king of routing!