Launch HN: Vouch (YC S19) – Business Insurance for Startups - sam-at-vouch We're Travis and Sam, co-founders of Vouch Insurance (https://vouch.us).
Vouch is a new insurance company that provides business insurance for startups. We make it easy for founders to get all of the business insurance they need as they build their companies -- ranging from basic coverage for business property through to more complex coverages such as Directors and Officers (D&O), Errors and Omissions (E&O), Cyber, Employment Practices Liability and a range of others that companies need to close financings, scale their teams, do deals and take on office leases. Moreover, these policies protect against painful but all too frequent risks that we all face as we build our businesses.
We started Vouch after seeing how painful it is to get insurance the old way, and also experiencing many of the risk events that all-too-often happen as one scales a company. Previously I co-founded and led the U.S. arm of Funding Circle, a role I held up until our IPO last year; through this experience I had to purchase insurance many times -- and each time, the experience was slow, painful, opaque, and paper-based. Travis previously worked at Silicon Valley Bank and say that pain-point across their portfolio and, at the same time, saw the interesting developments in other insurance sectors -- Root, in auto insurance, for example -- and similarly believed that there must be a better way for startups to buy and manage their insurance.  Although there's been a lot of work done on the distribution side of insurance (better agent and brokerage-type businesses) no one has really tried to build better insurance products tailored to meet the needs of startups.
Traditionally, a founder needs to go through a broker to purchase the coverages we offer directly, and that process can take weeks, or even months (it took us over 60 days to get the E&O coverage we needed at Vouch to start writing business, as one ironic example). We designed the Vouch experience so that a typical company can get everything they need in under 10 minutes and with 0 paperwork. We also tailored the insurance products we're offering to meet the unique needs of technology startups. As a result of this, our hope is that the entire experience should feel more tailored, and basic policies are quite affordable -- as little as $200/year. Our basic package  is 30% cheaper than anything else we've seen in the market and on average we're seeing our members save ~13% relative to other options.
We started the company after years of experience building and investing in other companies. Our view is that building a company is hard enough -- worrying about risk management and insurance shouldn't be, and we're building Vouch so that can be the case. We're now live in eight states, and are excited to say that we can now serve companies based in CA -- a state that's home to many startups. We'll be in many states all across the country shortly, as we want to help entrepreneurs wherever they're building great technology companies.
If you have questions about business insurance or any current insurance needs we'd love to connect. And we'd welcome reactions, feedback and any of your experiences around business insurance. Thanks for reading and if of interest check us out at: https://vouch.us <comment><AUTHOR> I just submitted an application, and have been chatting with Myles, who has been very helpful and responsive.

Vouch is not able to proceed with my application because I run a bootstrapped business – zero funding and no debt. Vouch's current underwriting guidelines require at least $150,000 in funding, which seems odd.

I'm not sure why this is the case - if anything, my business has a much lower risk profile, since I have fewer counterparties, and don't have the exposures that would necessitate D&O, EPL, EB, or FD.

I hope this is something that Vouch will discuss with the reinsurer. I've been looking for a service like this for years now, and I'd like to vote with my dollars.

<comment><USER2> This is also something you see with some other corporate products for startups, for example, Brex explicitly says "Credit limits are based on the cash you've raised and/or equity in your company" [0] while Stripe instead asks about revenue [1] (I assume because they're in a unique position to verify it). I guess different b2b companies are looking to serve different kinds of businesses (shocking conclusion, I know).

All of that said I definitely agree, as an outsider, it seems like a bootstrapped business would have a lower risk profile, but a venture-backed business would have faster growth potential and thus be worth more in premiums, so maybe that's their reasoning?

[0] [https://brex.com/startups/](https://brex.com/startups/) [1] [https://stripe.com/corporate-card](https://stripe.com/corporate-card)

<comment><USER3> I'm not sure that bootstrapped businesses have a lower risk profile, at least depending on what the other business views as risk.

VC backed companies have a route to capital and lines of credit if they tank. What is Brex/Vouch going to do if your bootstrapped business tanks? Take your house?

<comment><USER2> That's a good point, and bootstrapped businesses (in the US at least) are generally incorporated in a manner that protects personal assets, so the provider would not even be able to go after the owner's house in most cases.

</comment></comment></comment></comment><comment><USER4> One of the selling points of working with an insurance broker is that they go out and get quotes from multiple insurers, and then you compare the various policies + prices and pick the best one. (Also why the process is sometimes slow)

It looks like you're taking a very different approach of not comparing options from outside insurers and working with a single insurer instead. Why is it better to go through you than a more traditional broker (ie. [https://foundershield.com/](https://foundershield.com/))?

<comment> sam-at-vouch My experience has been that brokers can add a lot of value with mature companies, but they oftentimes over-complicate things for earlier-stage companies -- so making things simple, transparent, etc. is really key, which is what we're trying to do at Vouch. In addition, brokers sell insurance products that aren't tailored for start-ups. As a result, a lot of companies are either over or under-insured. And, on average, we're seeing our pricing as being very competitive relative to incumbents' offerings. We're not closed off to offering our programs through brokers, but for most start-ups we think this adds an unnecessary layer of cost/inter-mediation.

<comment><USER5> I build software for a similarly-disruptive insurance-selling company that has evolved from aspiring to replace brokers to collaborating with them. It took us at least 5 years to fully reverse our strategy from being broker- antagonistic to broker-collaborative. It has done wonders for our business. Excuse my rant, but I hope somebody finds it helpful!

I'd offer a strong word of caution against the attractive idea of viewing brokers as rent-seeking middlepersons. As a seller, you are probably very well-informed on the relative quality of your product. From that perspective, it's a simple mistake to assume that consumers have better information than they do, and thus don't need a broker.

The reality is that insurance (just like any kind of financial instrument) is extremely complicated. Brokers are a individually-tuned filter of information, telling clients exactly what they need to make an informed decision. In the insurance industry, consumers can often get this value at no cost to them (the insurance carrier pays the commission). Premiums are priced identically, whether you purchase through a broker or not; so, as a consumer, why _not_ choose a broker?

To help people (investors?) escape this mindset, I like to compare brokers to Certified Financial Planners. To the informed, CFPs seem like an embarrassing inefficiency in a marketplace that's vibrant with free information. In reality, CFPs provide a real value to a large segment of the population: decent returns, accountability and agency, a human to soothe them through tough times, and a filter for information that feels irrelevant to daily life. Most people want to think about insurance exactly once per year, for as little time as reasonably possible, and no COO got fired for purchasing insurance through a broker.

Bottom line, if your value prop to investors is selling directly to avoid commissions, you're selling them on:

(1) your ability to convince consumers that your information funnel is a better UX (arguable)

(2) your customer support is more versatile than a broker's (unless you're open to encouraging customers to go to a competitor, this is impossible)

(3) while you're accuring brand value to satisfy (1) and (2), you're comfortable with having a growth curve with a low ceiling.

Instead, I'd recommend seeing brokers as external salespeople. If you make a product that is easy to explain and sell, and develop a first-class broker experience (receiving commissions, viewing book of business, CRMing with clients), they will actively convince their clients to switch to you at that one time each year. Brokers already have a rapport of trust with their clients, that _you_ can leverage, even as an upstart, to win market share.

More concretely, in our experience brokers can convince people to change their carrier more effectively than world-class marketing. I'd recommend visualizing your commission spend as marketing spend. Commissions and broker tooling are levers you can pull to grow faster and higher.

It's a harder-to-sell vision, because investors tend to see brokers as an economic inefficiency to be "disrupted" away. But, again in our experience, it's the only practical way to bootstrap some table stakes in the market.

Anyways, hope whoever else reads this finds this insightful. Just my two cents; I think you all have a great vision, and I wish you the best! Learn from our hard-earned mistakes, please :)

<comment> sam-at-vouch Thanks for the thoughtful post -- let me expand on what I wrote previously, as I want to be clear: our goal is not to disrupt brokers; rather, it's to build new insurance products that uniquely meet the needs of high-growth technology (and over time, other entrepreneurial) companies. Our view is that brokers are very valuable for some clients -- and for certain other companies, especially very early stage ones, it's more hit or miss -- my experience is that's the case because it doesn't make sense for most brokers to serve small commercial accounts. We're very open to working with brokers over time for some clients/products/etc. Thanks again for the thoughts.

<comment><USER6> Thanks for expanding. Brokers that work in brick and mortar have their place. Based on reading about your company I would make the pitch that the customer wouldn't hire a defense attorney to incorporate your company (though you could and not that they wouldn't do a good job) and likewise you wouldn't hire silicon legal to defend you in a criminal trial. The last thing you want would be a broker or ultimately an underwriter that is unfamiliar with your business model that fights tooth and nail with you on simple stuff that the Hartford would take care of in a heartbeat.

</comment></comment></comment></comment></comment><comment><USER7> It would be interesting to know a few technical/regulatory details.

1- You say you're live in 8 states. Which ones? 2- Did you have to obtain 1 license per state? 3- How much time did it take to obtain each license? What it entailed? Was a prerequisite being a US citizen or being a US company? 4- are/will your employees that deal with the sales legally required to be individually licensed agents? 5- "Pricing depends on a variety of factors including policyholder location". Could you elaborate on the regulatory aspects, if any, of this?

<comment> sam-at-vouch Hi -- thanks for the questions, to each: 1) we're live in CA, OR, UT, CO, IL, MI, OH, IN and MI (so 9 states); we'll be launching in many more over the coming months; 2) insurance licensing happens at three levels -- for the entity (i.e. our insurance subsidiaries), for the program (the insurance products we've launched) and for people involved in selling/distributing the product. 3) Program approvals range from a few weeks to up to 9-12 months, depending on the state (in the U.S. insurance is regulated primarily by the states). There's no requisite to be a U.S. citizen to get insurance licenses, at least as far as I'm aware; and 4) yes, all of our people involved in selling (or in insurance-speak "producing") are and will be licensed.

<comment><USER8> Just a heads-up that you've listed MI twice here.

<comment> sam-at-vouch Sorry for typo -- will amend!

</comment></comment></comment></comment><comment><USER9> Who provides the underwriting? (I'm assuming you do not have anywhere near the capital to do it yourselves)

Have you paid any claims out yet?

In my experience these insurance products (ex: E/O coverage for bespoke software services) are a checkbox on a requirements doc for dealing with large companies. They're never invoked and it ends up being a private tax. All the customers care about is if you have coverage and if the underwriter has a minimum rating or is from a preapproved list.

<comment> sam-at-vouch We worked closely with our reinsurance partner (Munich Re) to develop our underwriting and pricing framework. They are back-stopping any claims associated with our policies, and are one of the two largest and best-rated re-insurers in the world.

We haven't paid out any claims yet -- our oldest polices are only about three months old -- but have built out a claims management approach to ensure we're doing the right thing for our members (while also of course making sure there isn't claims fraud).

In terms of where these policies are needed -- you're right, in some cases business insurance is a requirement to close contracts (E&O and Cyber are frequent contract requirements) but in my own experience other lines are really valuable. I've seen meaningful property claims in other businesses I've been involved with, and there are good reasons to have EPLI and D&O as well.

Thanks for the comments.

</comment></comment><comment><USER10> Thanks for the response to my other question! I have another though

Your filing indicates a 10% discount for a data sharing agreement. Can you expand on what data my business would share with you to be eligible to get that discount?

</comment><comment><USER11> I'm not really certain what void this is filling. As a solo-entrepreneur, it took me less than an hour all told to get Hiscox insurance. What is the competitive advantage to this? Why would I choose this over the alternative?

<comment> sam-at-vouch I'm glad you had a good experience for your business, but I can also say that's not the case for many entrepreneurs we've worked with (we've done coverage assessments for 150+, and that's growing quickly). With Vouch in ~10 minutes you can get access up to 10 coverages, and I'm not aware of any other platforms that do that. In my experience if you need anything more sophisticated than coverage for GL and personal property you'll probably be looking at a process that takes a few weeks at least (and I've seen it take 2 months+ for some of the coverages we offer). Lastly, because we've tailored our insurance programs for tech start-ups, in most cases we're cheaper -- our basic coverage starts at ~$300/year, and in general we're seeing companies save 14%/year.

</comment></comment><comment><USER12> If nothing else, many startups that have raised seed and A rounds don't have the proper insurance. It's typically because many founders are uninitiated and don't see the value and aren't required to get it.

I couldn't imagine anything more important, from a protect my investment POV, than making sure a bunch of no significant corp experience founders / execs have the proper E&O coverage (among other coverage).

<comment> sam-at-vouch Thanks for the feedback and very much agreed: for many companies, E&O is a critical coverage and one that founders may not know much about. It's a core part of our offering for companies that have launched product and are scaling. Thanks again for the thought.

</comment></comment><comment><USER13> I think a missing opportunity is just content on what startups typically need and what each insurance covers without needing to sign up e.g. add some breadcrumb nav flow.

The font on the "startup insurance" page with the information on each type of insurance is too small and looks like fine print rather than essential information for anyone looking to see what your product does or offers.

<comment> sam-at-vouch Thank you for the feedback. We agree -- education is key here. Thanks also for the feedback on the font; we'll see what we can do to make that better/easier to read. Thanks again.

</comment></comment><comment><USER14> I work doing adverting for an insurance brokerage full-time and everything is lacking any obvious principle of organization.

I feel this is partly due to the 1-to-1 selling style of most insurance salespeople... it moves like molasses in a digital world.

Another issue is the wholesale price. You need to be selling a lot of a particular type of insurance to get the best pricing for your clients.

Having the market to place insurance is important. Many small commercial insurance brokerages end up having to send things to a "General Agent" who has more markets then you do but marks up the policy higher so now you are not as competitive.

Advertising is tough, very competitive. Especially now with all of these funded to the hundreds-of-millions companies like eBroker and others coming on board.

I went from sending about $18-$22 per click for commercial insurance keywords on Google Adwords 3 years ago to upwards of $28-$35 per click now.

<comment> sam-at-vouch Thanks for sharing your experiences and agreed w/ the pace of change in insurance generally. Our view is that by building an insurance provider end- to-end using software we can take out a lot of this friction and better adapt to our clients' needs. Thanks again.

</comment></comment><comment><USER15> Getting insurance has been by far one of the easiest back off tasks we deal with (Hiscox / Chubb). The upsides you offer seem fairly low compared to having a low trust insurance vendor. Do you have an insurance rating?

<comment> sam-at-vouch Thanks for the question: our policies are backed by Munich Re, which is an A+ rated (by AM Best) reinsurer.

</comment></comment><comment><USER16> Your questions around which types of companies you insure are very narrow - are you only focused on a specific set of startups or are you able to insure more broadly yet?

<comment> sam-at-vouch Right now we're focused on providing insurance for technology start-ups (so think software, internet, fintech, digital health, etc) but we're planning broaden this so we can serve a much wider range of companies over time. Thanks for the question.

</comment></comment><comment><USER17> Can you serve a company "based" in California in that all employees are here, even though it's a Delaware C corp with foreign registration in CA?

<comment><USER18> Same question for Delaware corp based in NYC.

<comment> sam-at-vouch We're sadly not live in NY yet but will be soon -- I'll let you know as soon as we are.

</comment></comment></comment><comment><USER10> 1\. Is your pricing publicly available on System for Electronic Rate Filings and Forms (SERFF)?

2\. Are you justifying your pricing by saying "me too" and copying an existing competitor's pricing? If so, which competitor is that? If not, how are you justifying your rates?

3\. Are you hiring for software/product?

<comment><USER19> It (or at least the BOP part of it) is filed [1] in CA as STNA-132004682 (State National is the fronting carrier).

Filed Loss Ratio (including LAE) is 65%. Basically ISO rates and forms with exceptions...they also copied some Chubb stuff.

You can read the more interesting parts in this exhibit [https://docdro.id/R1KyTdL](https://docdro.id/R1KyTdL)

Looks like the major play here is that ISO has very little pricing granularity for tech companies (they are classified as "Offices (Not Otherwise Classified)") and that Vouch will focus on more differentiated pricing (based on policyholder sales, not as much on Limits).

D&O and E&O will be interesting...will look for that later...

EDIT: So that's STNA-132007333, and there they copied Rates and Rules from Great American, and use their own forms (which I'm guessing are somewhat similar as well).

[1] [https://filingaccess.serff.com/sfa/home/CA](https://filingaccess.serff.com/sfa/home/CA)

</comment></comment><comment><USER20> So you are insurance aggregator or facilitator?

Who insures you in case of a big claim that exceeds your equity?

A calamity like say the Exxon Valdez?

A great service but startups will pony up in good faith, but what happens when the chickens come home to roost?

<comment> sam-at-vouch We are an insurance program manager (or "MGA" in insurance speak), and our insurance programs are back-stopped by Munich Re, one of the largest re- insurers in the world. We very purposefully didn't set the company up as traditional broker, because then we'd be reliant on a bunch of old-fashioned and off-line partners, which would undermines the experience we're trying to create for our clients.

</comment></comment><comment><USER21> We use NewFrontInsurance. I've also seen companies like Cover.

How do you guys compare?

<comment> sam-at-vouch Both NewFront and Cover are digitally-enabled brokers -- they distribute existing carriers' insurance policies. We think highly of both of them, but we see a need to create insurance programs (including the digital experience and the underlying insurance products themselves) that will work well for early- stage technology companies, which is our focus.

</comment></comment><comment><USER22> In what sense are you "launching" today? In the news, you raised a 45m Series B recently. According to LinkedIn, you've been at it for 18 months already. So what's new?

<comment><USER23> "Launch HN" just means launching on HN. YC startups each get to do it once.

</comment></comment><comment><USER24> This is a BIG space open for changes.

Technically this is not a new entrant, just a different way to recycle capital. Munich Re (one of the top three reinsurers in the world) will provide reinsurance/capital to Vouch or anybody who could bring growth to them.

The post is a bit misleading. A lot of startups in that field use the standard bashing of "middlemen", "making the process more efficient", etc when in fact they are just another broker. Just about EVERY startup in that field uses the same language (Lemonade, MetroMile, etc…).

In this case Vouch is an agent of Standard insurance, a subsidiary of Markel, a company in the business of providing "Fronting" for large reinsurers: [https://www.statenational.com/fronting/](https://www.statenational.com/fronting/) [https://interactive.web.insurance.ca.gov/webuser/Licw_Agy_De...](https://interactive.web.insurance.ca.gov/webuser/Licw_Agy_Det$.STARTUP?Z_ORG_ID=386926&Z_AGY_LIC_NBR=0M81262)

An analogy would be emails: you can go directly to Gmail, Yahoo or whatever or you may access all those emails on your Mac Mail or whatever . What matters is the provider behind (their tech, spam capabilities, etc.)

Vouch is an agent of Standard Insurance, not really a top player in the field of insuring tech. Chubb and Hartford are.

Vouch is thin on experience with only two endorsed agents (I assume the founders) who only got their license one year ago: [https://interactive.web.insurance.ca.gov/webuser/licw_endors...](https://interactive.web.insurance.ca.gov/webuser/licw_endorsees$.startup?Z_ORG_ID=386926) And have no classes/credit or whatever to back up the fact that they can "provide advice" to startups: [https://interactive.web.insurance.ca.gov/licensestatus/licst...](https://interactive.web.insurance.ca.gov/licensestatus/licstatus?event=checkLicenseStatus&function=getCEDetail)

This is public information. The fact that they are licensed surplus lines brokers is a flag as they may place your business with "non-admitted insurers" instead of established insurers in your state (who have experienced claims people, insurance wording approved in your state, and backed by state guarantees). "Non admited insurers" is an opaque field of various insurers, some are good, some are just shabby. Just be aware.

Most of the time you will be better off with admitted insurers (Hartford, State Farm, Chubb) for a number of reasons: they have the experience, they pay claims promptly, and are cheaper. The state will keep an eye on them if they don't pay claims promptly.

I assume that Vouch will also farm out the claim process to contractors. I just don't know.

I am not saying that Vouch is good or bad, just be aware, don't buy into the BS and be an informed buyer.

Buying from a mid-size insurance broker provides some piece of mind: they know what they are doing, have the equivalent of a large enterprise team dealing with big, publicly traded tech companies, and they can call on those in house folks to help if needed or if you have a claim that the insurance company "initially" denied. They can apply "gentle pressure" because they have buying power.

Of course if you deal with Vouch, an agent of Standard Insurance, your options are limited if you get in trouble.

You may think "I will sue them". You don't sue insurance companies, or very rarely (and only if you are a consumer, not a company) because they have in house counsels they pay $50/hr against your $1,000/hr lawyers.

<comment><USER19> I think you're a bit too pessimistic/cynical...

1\. It's State National, not sure where you got "Standard" from.

2\. Vouch is acting an MGA/Program Administrator, including being involved in underlying insurance product/actuarial design. This is WAY different than being a regular broker/agent. (It's essentially the difference between sales, and sales and operations.)

3.State National doesn't have a reputation for _anything_ \--that's what makes them a fronting carrier! Besides providing the paper/AM Best Rating/filing assistance they are not involved in the day to day.

4\. Being Surplus Lines _Licensed_ is not a red flag; if they want to offer (possibly in the future) additional options for hard to place risks, then it's being used appropriately. If they are using it just to be able to offer "something" when they know, say, Hiscox would be happy to cover them, then I'm not enthused.

5\. Having only a few licensed producers on stuff is not unusual for now; they may be sharing staff with a partner/TPA, plus they may have a large inhouse underwriting team. With so many coverages being offered though, they do need a larger staff than normal to be proficient.

6\. It's quite likely that being focused on a narrow class means they can be more responsive/flexible and better able to price risk. I assure you the Hartford does not care at all about the VC funded tech startup market. Program business can be very lucrative AND beneficial to those being covered. Some brokers (large or small) have NO IDEA how to handle tech startups; if Vouch proves itself they may eventually try to place business with Vouch.

7\. Some insight in how they will handle claims would be good, I agree-- ESPECIALLY for D&O and E&O where legal assistance is critical (and specialized). Being able to offer/get your filings approved for many different coverages is only a piece of the puzzle...you need experienced people on the claims side, irrespective of the bare ability to "pay claims". The fact they don't even discuss claims handling on their site makes me sad :-(

8\. Oh, insurance companies get sued all the time for denying claims, not that you want to be in that position...

PS You founded Esurance? Cool!

<comment><USER24> Thanks for the clarification. Sorry about sounding cynical, that was not the intend. I have just seen a lot of BS over the years and I super careful trying to understand what is behind that smokescreen. Good luck with the venture!

<comment><USER19> Oh, I'm not affiliated with Vouch! Just an interested observer...

</comment></comment></comment></comment><comment><USER25> Interesting!

</comment><comment><USER26> i see they are trying to make a derivatives market for startups...

bubble isn't growing fast enough... need more leverage...

someone build a longer 2x4

<comment><USER23> " _Please don 't post shallow dismissals, especially of other people's work. A good critical comment teaches us something._"

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER15> He could have phrased it better, but reading the initial post reminded me a bit of banks trusting AIG to insurance their high risk investments.</comment></comment></comment>


Who Owns Kafka? (2011) - <AUTHOR> https://www.lrb.co.uk/v33/n05/judith-butler/who-owns-kafka <comment><USER2> > What no one could have predicted, however, is that a trial would eventually > take place after Esther's death in which her daughters, Eva and Ruth, would > claim that no one needs to inventory the materials and that the value of the > manuscripts should be determined by their weight – quite literally, by what > they weigh. As one of the attorneys representing Hoffe's estate explained: > 'If we get an agreement, the material will be offered for sale as a single > entity, in one package. It will be sold by weight … They'll say: "There's a > kilogram of papers here, the highest bidder will be able to approach and see > what's there." The National Library [of Israel] can get in line and make an > offer, too.'

That's a very strange auction strategy. You would think that a proper cataloguing would increase the amount you would earn, and it could be done by a hired gun for cheap (since Kafka experts would clamor for access). From a game theory perspective, refusal to disclose contents should, by the 'unraveling' argument, drastically lower the value of the papers to rational bidders since it is a strong indication that the contents are worth a lot less than prospective bidders previously estimated before the refusal (since if the contents were as valuable or _more_ valuable, the owners would have incentive to reveal to ensure the price or raise it; absence of evidence is evidence of absence).

One possibility is that they are counting on bidders being irrational and desperate, in which case refusing to disclose keeps valuations irrationally high and exploits the winner's curse; possibly the strategy here is to make the Israeli National Library overpay because it is committed by ethnonationalism to pay whatever price necessary, and revealing the contents risks either public criticism of the National Library overpaying or scaring away other bidders as they learn the papers are boring & losing leverage over the National Library.

</comment><comment><USER3> >In 1988 she sold the manuscript of The Trial for $2 million, at which point it became clear that one could turn quite a profit from Kafka

To some, $2 million would be a bargain for, say, a new Kafka novel.

The conspiracy theorist side of me has often wondered whether maybe Kafka did it intentionally. Intentionally left his novels unfinished, intentionally started a legend that he requested them to be destroyed unpublished, etc. It would certainly resonate with the tones and themes that permeate his works. Perhaps he himself realized that no ending could possibly suit a novel like "The Castle" better than the unfinished lack-of-ending he gave it, which makes our hearts yearn so strongly for an ending, and which is such a great parallel with the futility of the novel itself.

<comment><USER4> You have it exactly backwards; Kafka was one of those artists who can only perceive the flaws, not the qualities, of their artwork. He was ashamed and wanted to be forgotten by history.

</comment></comment><comment><USER5> For the curious, this trial was eventually resolved in favour of Israel's national library, on the grounds that this was Max Brod's instructions in his will:

"But Judge Kopelman Pardo rejected Ms. Hoffe's claim that the papers were a gift from Mr. Brod to her mother, instead viewing them as a trust she was to administer. The judge noted that Mr. Brod's 1948 will instructed that his archive go to a "public Jewish library or archive in Palestine," and that he later specified Hebrew University, where Israel's national library is housed."

[https://www.nytimes.com/2012/10/15/world/middleeast/woman- mu...](https://www.nytimes.com/2012/10/15/world/middleeast/woman-must- relinquish-kafka-papers-judge-says.html)

</comment><comment><USER6> Not very related to the article, but I'd like to get some opinions from HN community (if I may). I've read quite a few books and short stories of Kafka (I think all(?) his novels and some short stories like _The Metamorphisis_ , _A Report to an Academy_ ) but most of them were really struggling readings for me and some incomplete (I never got to the very end of _The Trial_ ). His short stories, I read cover to cover in one sitting, and they were very enjoyable readings. I also read _Amerika_ (his first novel) cover to cover pretty fast and it was very enjoyable too. His other works, I find very challenging for reasons unknown to me. Kafka has a certain literary style that feels very enjoyable, reading it gives me so much joy, so it's definitely not that I didn't like them. It's also definitely not boredom, some Kafkaesque "exaggeratedly long" scenes were pretty interesting and reminds me of Tarantino (not that there is any direct artistic resemblance, it just gives me similar type of enjoyment). Deeper ideas argued in novels are also interesting, and I like reading other authors trying to express similar ideas. But overall, reading Kafka is for some reason really hard for me. It takes too long relative other books I read, I lose focus very easily and it eternally feels like I'm missing some pieces of both the plot and the artistic structure etc... I also read authors that have been major inspiration to Kafka like Dostoevsky and never had the same problem. Did anyone else have this problem? I'd like to hear some tips from literary folks here to help me read Kafka better.

EDIT: As absurd as it sounds, I had similar feelings when reading his letters to Milena. I read that book twice, it's one of those books I really like but I had similar problems mentioned above.

<comment><USER3> He only did three novels, so you've already finished 1/3 of them.

The Castle: The only difficulty here is some conversations take forever (probably because they were never revised). I'd suggest just plowing through them the first time, your eyes might glaze over and you'll miss stuff in them but it's ok, you can pick more stuff up on later readings. I've read The Castle many times and I still pick up new stuff from it.

The Trial: There's really only one chapter that's difficult, the penultimate chapter set in the cathedral. You could literally just skip it, if you're having trouble with it. You'll miss some self-contained goodies like "Before The Law", but you can always come back later. It has been said that except for the first and last chapters, most chapters in The Trial can be rearranged and read in whatever order you like. I seem to recall someone even created some sort of physical version of the book where you could literally swap chapters around.

<comment><USER6> The Castle, I really liked, much more than The Trial, but I couldn't finish that either. I don't exactly recall where was I stuck but I remember literally struggling to read as if studying Algorithms or Machine Learning. I read it both in English and German with similar difficulty.

The Trial was the only thing I read from Kafka that I found kinda meh and boring-ish, again made it a bit more than half way. I tried reading The Trial at least 3 times, maybe more, with same faith every time. (I eventually learned its ending in a literature class, but given other works of Kafka, it was very predictable). I'll give it a shot again and maybe skip chapters where I lose focus and come back later.

</comment></comment></comment><comment><USER7> Also interesting but unrelated: [https://en.wikipedia.org/wiki/Apache_Kafka](https://en.wikipedia.org/wiki/Apache_Kafka)

</comment><comment><USER8> At first glance I thought the answer was going to be LinkedIn...

<comment><USER9> Quoting from Apache Kafka's wiki page

> According to a Quora post from 2014, Kreps chose to name the software after > the author Franz Kafka because it is "a system optimized for writing", and > he liked Kafka's work Source: > [https://en.wikipedia.org/wiki/Apache_Kafka](https://en.wikipedia.org/wiki/Apache_Kafka)</comment></comment>


CuPy – NumPy-compatible matrix library accelerated by CUDA - <AUTHOR> https://cupy.chainer.org/ <comment><USER2> Not sure what is the motivation behind this library. There are already several array GPU accelerated array libraries -- PyTorch, TensorFlow, ArrayFire, it even looks like pycuda has a small array class.

<comment><USER3> Chainer and potentially CuPy (which was extracted from Chainer to be independent) were around before PyTorch as it served as inspiration for PyTorch. I feel like that's a good motivation for diversity in packages and ecosystems regardless of your feelings otherwise.

Along with a colleague I used CuPy in first Chainer and then PyTorch for implementing the Quasi-Recurrent Neural Network (QRNN) which at the time was far faster than even NVIDIA's optimized cuDNN LSTM whilst getting the same (or better) performance for many tasks.

CuPy at the time was both the easiest and most Pythonic of potential solutions for that problem - even if it did involve writing CUDA in Python strings =]

n.b. Our use case was literally pushing state of the art in research - CuPy is even more Pythonic if you're hitting more standard use cases.

[1]: [https://github.com/salesforce/pytorch- qrnn](https://github.com/salesforce/pytorch-qrnn)

<comment><USER4> PyTorch is almost (or even literally?) a fork of chainer, which can be seen when comparing example code. The latter was much more stable than the former for quite some time after PyTorch gained big popularity through Facebook. We have been using chainer for a lot of published NN research projects and only recently moved to PyTorch because students complained that they feel they can't put the more popular framework on their CVs..

I continue to have more sympathies for chainer.

<comment><USER5> >PyTorch is almost (or even literally?) a fork of chainer That's funny, I would have assumed PyTorch to be, like, the python version of Torch?

<comment><USER6> The PyTorch tensor library was originally basically the Python version of Torch 7. It's now moving closer towards NumPy's API (and farther from Torch 7).

Th autograd library was inspired by Chainer's design and took a lot of concepts (but not code) directly from Chainer. The neural network API is a bit of a hybrid. It's built on top of the autograd library but the layer names, implementations, and some conventions were inherited from Torch 7's NN and cuNN libraries.

(EDIT: and the name "autograd" originates from HIPS autograd library, which I think predates Chainer)

</comment></comment></comment></comment></comment><comment><USER7> I'm working with Numba's CUDA API and it works well as a drop in replacement for embarrassingly parallel functions.

<comment><USER8> I've done a fair bit of C++11 for CUDA and I was so happy to throw everything out and switch to Numba. It has some rough edges (like incomprehensible error messages when the type inference goes wrong) but it's been a pleasure overall to work with.

<comment><USER9> I've done a fair bit of Numba CUDA and I was so happy to throw everything out and switch to C++.

NubaCUDA gave me lots of small problems and a few big ones. The poor support for debug/perf tools and poor integration with other high-level python CUDA code (FFTs in particular) sent me packing, but the number of small problems was excessive in comparison to the size of my code. I had 5 reduced bugs at the bottom of my notebook and two paragraphs of "baggage" at the top to support a tiny little 50LoC kernel: one paragraph for the environment variables and one for patching nubacuda itself for a trivial API incompatibility that hadn't been fixed for the better part of a year. All of this for a tool that provided a diminutive subset of functionality at the intersection of both python and C. I've felt more computational freedom writing BASIC on my TI-83.

CuPy could well have changed that equation!

> incomprehensible error messages when the type inference goes wrong

NumbaCUDA is truly the galaxy-brain of type checking: first it complains loudly so as to force you to provide type information, then it opts to not complain about a mismatch, and _then_ it silently reinterpret_casts a double* to float* behind your back.

I know it's free software and I have no right to complain, but I sure sunk a lot of time into this dead end and regret it.

Spiffy icon though.

<comment><USER10> What's the difference of NuMBA CUDA and Pytorch or similar?

If you're doing custom kernels you should take a look at the Julia library CuArray [1] and generic kernels [2]. I really like that I don't have to dig into C++ and deal with all of the memory and kernel management.

1: [https://github.com/JuliaGPU/CuArrays.jl](https://github.com/JuliaGPU/CuArrays.jl) 2: [http://mikeinnes.github.io/2017/08/24/cudanative.html](http://mikeinnes.github.io/2017/08/24/cudanative.html)

<comment><USER9> My impression was that pytorch focused on linear algebra / deep learning. The reason I was playing with numbacuda in the first place was because part of my problem did not fit nicely into a (dense) linear algebra framework, so numbacuda's custom kernel support seemed attractive. Does pytorch have a good low-level kernel library? Or sparse linear algebra library?

I love Julia, but I haven't managed to convert anyone else on my team and I already spent my informal exploration budget for the GPU project on nubacuda, so JuliaGPU will have to wait for another time. I'll be sure to keep it in mind, though!

How is the CUDA debug/perf story with Julia? Does it play nice with the nvidia tooling?

<comment><USER10> Ah, that makes sense. I've only dabbled a little with DNN's recently, but pytorh/tensorflow seemed very targeted toward deep learning. Generic tools seem more useful to me. What are you doing with fft's?

I haven't dug too deep with CudaNative / Cuarray to understand the state of Julia perf debugging. Though here's one post on the topic:

[https://discourse.julialang.org/t/cudanative-is- awesome/1786...](https://discourse.julialang.org/t/cudanative-is- awesome/17861)

In general It's been very pleasant experimenting with gpu programming in Julia. I couldn't quite grok tensorflow code, and it's cool to just declare a Julia array and send it the GPU.

</comment></comment></comment></comment></comment></comment><comment><USER11> The problem I see with trying to emulate NumPy with a GPU accelerated version is that the communication overhead to the GPU is so high that you are losing a lot of performance as opposed to something like TensorFlow.

It takes a couple of microseconds just to start a kernel much less the time it takes to transfer data back and forth.

<comment><USER12> You don't transfer back and forth if you use managed memory.

</comment></comment><comment><USER13> Is it possible to pass cupy data to C? Possibly by accessing the array pointer.

I've used PyCUDA for a very long time. You can use Cython/ctypes/cffi to pass PyCUDA arrays to standard C/CUDA code.

</comment><comment><USER14> I use pandas mostly, rarely dropping down to NumPy. Are there non ML/neural network use cases where this library is meaningfully faster than numpy?

<comment><USER15> Anything that involves computing dot products on large matrices will be dramatically faster with cupy than numpy (depends on your graphics card of course).

</comment></comment><comment><USER16> How does this compare against Google's JAX?

</comment><comment><USER17> Cool I wish that other libraries like tensorflow could get cudnn installed automatically like this library!

<comment><AUTHOR> pytorch does the same:

[https://news.ycombinator.com/item?id=15622763](https://news.ycombinator.com/item?id=15622763)

<comment><USER18> At least up until 2 months ago, you had to do it manually, i don't know if that changed.

<comment><AUTHOR> When I installed pytorch, I had already installed CUDA and CUDNN (as I needed them for tensorflow) so I have not verified what smhx said.

</comment></comment></comment></comment><comment><USER19> In another thread I ask about pyiodide ... is this an answer to use cuda in browser under python?

<comment><USER20> No, browsers only expose specific Web APIs and CUDA is not one. WebGL doesn't even provide access to compute shaders.

</comment></comment><comment><USER21> Can you use Pandas with it? I'm guessing.. no?

<comment><USER22> Maybe not immediately, but isn't this what Pandas' new array interface[0] should facilitate?

[0] [https://tomaugspurger.github.io/pandas-extension- arrays.html](https://tomaugspurger.github.io/pandas-extension-arrays.html)</comment></comment>


Hans Ulrich Obrist, In Conversation with Julian Assange - <AUTHOR> http://www.e-flux.com/journal/view/232

<comment><USER2> After Mubarak fell, we witnessed an extraordinary change in rhetoric from Hillary Clinton and the White House, from "Mubarak is a great guy and he should stay," to "Isn't it great what the Egyptian people have done? And isn't it great how the United States did it for them?" Likewise, there is an idea that these great American companies, Facebook and Twitter, gave the Egyptian people this revolution and liberated Egypt. But the most popular guide for the revolutionaries was a document that spread throughout the soccer clubs in Egypt, which themselves were the most significant revolutionary community groups. If you read this document, you see that on the first page it says to be careful not to use Twitter and Facebook as they are being monitored. On the last page: do not use Twitter or Facebook. That is the most popular guide for the Egyptian revolution. And then we see Hillary Clinton trying to say that this was a revolution by Twitter and Facebook.</comment>


Kenyan tea workers file UN complaint against Unilever over 2007 ethnic violence - <AUTHOR> https://www.theguardian.com/global-development/2020/aug/01/kenyan-tea-workers-file-un-complaint-against-unilever-over-2007-ethnic-violence <comment><USER2> Executive Summary: Unilever uses multinational status to dodge litigation for failure to protect workers.

<comment><USER3> More like African community uses parent companies multinational status to file a complaint to UK courts and the UN instead of the local, Kenyan, court system. I don't think it's the only plantation which had to do such a thing, but since most would be locally owned they don't get dragged before the UN.

<comment><USER4> If a corporation is going to transplant workers to a plantation in an unsafe place, then the corporation has responsibility for keeping the workers safe.

A big element of their case that's oddly not mentioned in this Guardian article is that the violence was foreseeable and nothing was done to stop it. The tensions did not bubble up overnight, and this was not the first instance of violence in an otherwise peaceful place. There were many signs that something like this could happen, and Unilever declined to spend resources on keeping the workers safe. As a result many of the workers were raped and murdered. Now the survivors are seeking justice, and they deserve it.

Unilever are the corporation who was ultimately profiting from the employees' work, so the survivors have a right to seek justice from them directly.

<comment><USER5> > If a corporation is going to transplant workers to a plantation in an unsafe > place, then the corporation has responsibility for keeping the workers safe.

Citation needed - I never heard of a corporation being required (or allowed) to do law enforcement (private security companies might be contracted bt the government, but simply going and doing is most definitely not possible). The UN declares that a responsibility of governments, and one of the keys to the land claim and recognition of the state.

In _some_ jurisdictions it's legal to stop a crime by appropriate means, but it is not a responsibility of any private person or entity, and it also does not mean you do it normally - that's where you're crossing into the "private army" territory.

<comment><USER6> There ought to be some consequences for knowingly putting and leaving people in a place where they are likely to be raped and murdered. Contracting adequate (local or foreign) private security is not the only way to resolve a situation like this. You could also remove the people.

<comment><USER5> That's what the government is for. Either let the corporation be wholly responsible for safety (including things like surveillance, risk analysis, prevention etc), or don't complain about the corporation not doing it when there is a government saying they have it covered. Why are you not angry at the government for not stopping this foreseeable crime?

</comment></comment></comment></comment></comment></comment><comment><USER5> How can a foreign private company or person be responsible for protection of people during a civil war? Or what is the complaint about? The unpaid wages? Why is my unpaid wage not a human rights issue too, then - do I need to be african for that?

</comment><comment><USER7> My reading is that Unilever had a large non-indigenous workforce living on a plantation that was attacked. This was a western Kenyan plantation and I suspect the workers had travelled there from other west-Kenyan areas based on their tribe. I'm not expert on Kenyan law or the way these business contracts play out but I feel the issue at play here is if Unilever has a responsibility to protect workers when nationwide ethnic violence breaks out, and they have non-indigenous workers.

<comment><USER5> Are (foreign!) private people or corporations even allowed to operate a private law enforcement unit in Kenya? I thought the UN discourages that, and it's outright illegal in most countries of EU - meaning the corporation can't have such unit anywhere, that applies to its subsidies too; personal participation in such units is a serious crime as well.

</comment></comment><comment><USER8> I lived in Kenya before, during, and after the 2007 ethnic violence caused by the disputed results of the presidential election... but why is this on HN? It's completely unrelated to technology or startups.

<comment><USER9> The means we found out had some kind of electronic transmission so there's a bit of tech in it.

<comment><USER10> Actually, it's because Unilever hacked the corporate structure in order to dodge litigation. Legal engineering.

<comment><USER11> Corporate structures are often used to shield from frivolous lawsuits, like these.

<comment><USER10> From social engineering to societal engineering</comment></comment></comment></comment></comment>


Luminoth: Open Source Toolkit for Computer Vision - <AUTHOR> https://luminoth.ai/ <comment><USER2> Is this a reference to the Luminoth [1] from Metroid Prime 2?

[1] [http://metroid.wikia.com/wiki/Luminoth](http://metroid.wikia.com/wiki/Luminoth)

Edit - Turns out it is! From the github:

"Why the name?

The Dark Visor is a Visor upgrade in Metroid Prime 2: Echoes. Designed by the Luminoth during the war, it was used by the Champion of Aether, A-Kul, to penetrate Dark Aether's haze in battle against the Ing.

\-- Dark Visor - Wikitroid"</comment>


Ask HN: How much distributed tracing costs:Using OSS like Jaeger or paid product - <AUTHOR> Wanted to get your inputs on 2 solutions for tracing needs: 1. Open Source solutions like Jaeger - Is there a blog or link which analyses the cost of storage of traces to any of the backend like Cassandra or Elastic? Do you use any compression techniques before storing to DB?
2.Paid products like Datadog - Most of the vendors charge on a per-host basis. What's the underlying logic for charging like that?
If I were to use a vendor for this, I would like to be charged according to the number of spans sent for storage like in logs, say $2 per million spans or $0.2 per GB. Only Datadog seems to charge on such basis and asks for $1.7 per million of spans apart from $31 per host which covers 1M spans only. Does DataDog give you control of which spans to visualise & store?
- Does any vendor give you control of spans to process or a clear pricing estimate for the tracing part?
What other things should I look when using Jaeger or buying Datadog? My primary need is to monitor and debug my applications. <comment><USER3> Hey! I can't speak to the exact mechanism for pricing on a large or enterprise level account, but it sounds like you're just getting started with tracing - might be worth checking out Lightstep. Pricing is pretty transparent for different tiers of non-enterprise usage and it's seats+services monitored, NOT consumption, so there shouldn't be any end-of-month surprises. ([https://lightstep.com/pricing](https://lightstep.com/pricing))

Lightstep and several other vendors use open source standards so worst case scenario, you have great instrumentation even if you're not paying for a product right away. /shrug

(full disclaimer, I work at Lightstep albeit not in sales)

<comment><AUTHOR> Checked Lightstep. They charge 1900 USD for 8 services and 199 USD for each added services. So, if I have 15 service, it would be ~3300 USD/month - <USER2> seems a lot to me.

Also, don't understand why they are charging based on services? From that perspective, costing based on number of spans seems more transparent

</comment></comment><comment><USER4> The thing with Jaeger is you will likely need to deploy other services with it. For example, if you use Istio, pods need sidecars with non-negligible resource requirements. You'll need EFK / Prometheus.

You should include the salaries to deploy, debug, and maintain in the calculation. Not just Jaeger, all the supporting systems too. Finding people (you'll need more than one most likely) is a challenge as well.

You probably don't need distributed tracing at first, I have only seen a few people at this point. Start with basic logging and metrics if you have none today.

<comment><AUTHOR> Understand the total cost of ownership, etc. Was trying to understand the actual cost from running the infra etc. for Jaeger

<comment><USER4> It's not such an easy calculation, depends a lot on all the other things and the nature of your workloads.

<comment><USER5> Will it help, if I say 10 spans per request and 100 RPS?

Or even simpler, I use the dd-trace library and my application handles 100RPS

<comment><USER4> No, you need to talk about the running environment, the VMs, the platform, the other components, are your VMs in the same zone or region... There's a ton

Regardless Jaeger is a small piece of monitoring and logging. And an advanced capability I have not actually seen deployed. Sounds great, but often unneeded, at least until you have the other things in order

Have you thought about what happens if ingestion of logs breaks?

If your primary need is to monitor and debug, then don't worry about distributed tracing yet

<comment><USER5> Just being curious about the cost, why running environment or VMs, the platform matters? Region matters for data transfer costs that I can understand.

I can relate more with the number of spans based pricing because eventually spans are saved to DB apart from the cost of running Jaeger infra. Can you please provide more details into how the running environment, the VMs, the platform, the other components affect the cost? Will be helpful

<comment><USER4> All the components of a robust monitoring system take up space. Disk, CPU, Mem, developer time

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> Look at OpenTracing. This is the merger of two tracing initiatives and will provide the pointers to both commercial and open source solutions. [1] There is some excellent capabilities to instrument an existing java app through the jvm [2] with OpenTracing.

[1]: [https://opentracing.io/](https://opentracing.io/) [2]: [https://medium.com/opentracing/opentracing-on-kubernetes- get...](https://medium.com/opentracing/opentracing-on-kubernetes-get-yer- tracing-for-free-7a69cca03c8a)

<comment><AUTHOR> Yes, aware of that. Also, opentracing is now merged with opencensus to create a new project called opentelemetry. That is the upcoming standard now.

my questioning was more around running the cost of running these tools

</comment></comment><comment><USER7> Hello, I am a Civil engineering student in my final year of Bachelor's degree. I am very interested in the field of machine learning and ai.But i am trying to determine how can i integrate my academic knowledge with machine learning. Any suggestions or ideas would be greatly appreciated.Thank you.</comment>


Toapi – Let any web site provide APIs - <AUTHOR> http://www.toapi.org <comment><USER2> Now I don't want to be a downer: but we collectively seem to have forgotten that HTML as a markup language with sufficient semantic elements, is a perfect API in itself. In fact, if we had stuck with XHTML I would've postulated that it would've been an even better API than JSON due to XPath, XQuery and XSLT.

"HyperText is a way to link and access information of various kinds as a web of nodes in which the user can browse at will. Potentially, HyperText provides a single user-interface to many large classes of stored information such as reports, notes, data-bases, computer documentation and on-line systems help. We propose the implementation of a simple scheme to incorporate several different servers of machine-stored information already available at CERN, including an analysis of the requirements for information access needs by experiments… A program which provides access to the hypertext world we call a browser — T. Berners-Lee, R. Cailliau, 12 November 1990, CERN "

Web apps are somewhat backwards in my opinion. We completely lost the idea of a markup language and decided "wait, we need an API". So we started using JSON instead of the actual document (HTML or XML) to represent the endpoints. And then we patted ourselves on the back claiming "accessibility".

<comment><USER3> That was fine when (X)HTML was about semantics, but once it got used for presentation too, the contract was broken. I scrape a few sites (due to the lack of RSS feeds and the desire of UK organisations to treat Facebook pages as their 'News feed') and it's easy... until they do a redesign and the HTML changes.

For that reason alone, an API response format that stays constant no matter how the site looks appeals to me.

<comment><USER4> > but once it got used for presentation too

So everything after HTML 2.0 (RFC 1866)? The STYLE and FONT elements were added in HTML 3.2, but when tables were added to HTML 2 in RFC 1942 they already included presentational attributes. Heck, RFC 1866 already included the IMG element with an ALIGN attribute, as well as typographic elements like HR, BR, B, I and TT (for monospace text).

It sounds like you're being nostalgic about a time that never was, especially for XHTML (which the Semantic Web crowd loves to misremember as being 100% about semantics and not just a hamfisted attempt to make HTML compatible with the W3C's other XML formats).

<comment><USER2> Really people didn't like XHTML because they didn't want to close their elements. That's it. And now most web pages don't even parse in an XML parser. What elements are standard and which are not is completely arbitrary, what the browser does with them doesn't really matter either. What matters is that you can extract the data from it, if you know the structure (either by following some standard, or by having out-of-band documentation). In that respect JSON and X(HT)ML are similar, except now you can't scrape web pages with a single GET request from the canonical URI, and instead need to run a fully fledged browser that parses javascript and/or read some site specific JSON docs (if any are provided at all).

<comment><USER4> > because they didn't want to close their elements

That is an incredibly uninformed view of early 2000s web content authoring. The reason people didn't like XHTML was that it provided no tangible benefit.

In fact, my experience was quite the opposite: technical people _loved_ XHTML because it made them feel more legitimate, they just had to sprinkle a few slashes over their markup. Validators were the hot new thing and being able to say your website was XHTML 1.0 Strict Compliant was the ultimate nerd badge of pride.

But these same people didn't use the XHTML mime type because they wanted their website to work in as many browsers as possible.

> And now most web pages don't even parse in an XML parser.

Again with the nostalgia for a past that never was: most web browsers never supported XHTML, they supported tag soup HTML with an XHTML DOCTYPE but an HTML mime type. Why? Because they supported tag soup HTML and only used the DOCTYPE as a signal for whether the page tried to be somewhat standards compliant.

</comment></comment></comment></comment></comment><comment><USER4> As HN sometimes likes to pretend XHTML is the answer to all problems with HTML and the Semantic Web would have worked if only developers hadn't been such idiots at the time, let me reiterate a few things:

First off, a full disclosure: I was (and in principle still am) a big fan of web standards and was a strong believer in XHTML and XHTML 2. I thought XML was going to save us all.

Here's a hard truth about HTML: it was never about semantics. HTML was created to connect digital documents. Most of the initial tags were fairly ad hoc and loosely based on CERN's SGML guide, as is still evident from the awkward h1-h6 elements[0].

The biggest thing the first W3C spec[HTML32] added to HTML? The FONT element (and arguably APPLET because Java applets were a thing). Note that it really didn't add any elements for additional semantics. It was just another iteration on HTML that merged minor changes to HTML 2.0 with some proprietary extensions (mostly by Netscape).

HTML 4 followed shortly after and promised to rid the world of presentational markup because CSS had become somewhat widely supported (having won against Netscape's JavaScript StyleSheets, which rapidly faded into obscurity). It attempted to do this in a backwards compatible way by defining three different DOCTYPEs: Strict, Transitional and Frameset.

At this point it's worth emphasising what has become the most important mantra in web standards: Don't Break the Web. HTML 4 Strict was trying to enable a break as an opt-in. The website would behave exactly the same way but the DOCTYPE would tell browsers the page doesn't use certain elements the spec defined as deprecated.

Of course by this point web browsers no longer cared about SGML and were purpose built to handle HTML and be able to render whatever junk a content author would throw at them. The Browser War was raging (Netscape still believed it could make money selling commercial licenses for their Navigator -- yes, surprise, Netscape Navigator wasn't free for commercial use) and if your browser couldn't render a page but a competitor's could, that's where users would go.

So in practice Strict vs Transitional didn't make any difference except as an indicator to how sloppy the author likely was with their markup. Eventually Internet Explorer started using it as a shibboleth to determine whether to fall back to "quirks mode" or follow the standards more closely but deprecated elements would still render in strict mode and the only thing that cared were automated validators that spit out pretty badges.

When the W3C created XHTML[XHTML1] the entire point was to drag HTML into the XML ecosystem the W3C had obsessively created and that just wasn't getting much traction on the web. Instead of having to understand SGML, browsers could just learn XML and they'd be able to handle XHTML and all the other beautiful XML standards that would enable the Semantic Web and unify proprietary XML extensions and open standards side by side.

Of course the only flaw in that logic was that browsers _didn 't_ understand SGML. Adding support for XML actually meant implementing an additional language in addition to HTML and as XML had much more well-defined implementation requirements, this among other things meant that web pages that weren't well-formed XML would have to break _by definition_ and browsers were only allowed to show a yellow page of death with the syntax error.

This is why browsers for the most part ended up ignoring XHTML: Firefox supported XHTML but developers were upset with WYSIWYG tools breaking their websites in Firefox and because ensuring all your output is proper XML is hard, the easier fix was to just send your "XHTML" tag soup with the HTML mime type so Firefox would just handle it as HTML (which all the other browsers did anyway -- except Internet Explorer, which for once did the right thing and dutifully refused to render proper XHTML because after all it only understood HTML, not XHTML).

But as I said: XHTML wasn't about semantics. In fact, XHTML 1.0 copied the exact same Transitional/Strict/Frameset model HTML 4 had introduced, allowing developers to write the same sloppy markup (but now with more slashes). And the W3C even specified how authors could make sure their XHTML could be parsed as HTML tag soup by non-XHTML browsers (which led to millions of developers now thinking closing slashes in _HTML_ are "more standards compliant").

A few moments later the W3C created XHTML 1.1[XHTML11], which was mostly just XHTML 1.0 Strict but now split into multiple specs to make it easier to sneak other XML specs into XHTML in the future. Again, nothing new in terms of semantics.

Meanwhile, work began on XHTML 2.0[XHTML2] which should never see the light of day. XHTML 2.0 was finally going to be a backwards incompatible change to HTML, getting rid of all the cruft (more than XHTML 1.1 had tried to shake off by dropping Transitional, even promising to kill a few darlings like h1-h6) and replacing lots of HTML concepts with equivalents already available in other XML specs. XHTML 2.0 would finish the transition that XHTML had started and replace HTML with the full power of the XML ecosystem.

Except obviously that was not going to happen. Browser vendors for the most part had given up on XHTML 1 because authors had no interest in buying into a spec that provided no discernable benefits but would introduce a high risk of breaking the entire website or requiring workarounds for other browsers. Netscape was dead, Internet Explorer had stabilised and didn't seem to be going anywhere, the Browser War that had fueled innovation was largely over.

But even so, XHTML 2 wouldn't really have added anything in terms of semantics. That wasn't the goal of XHTML 2. The goal of XHTML 2 was to eliminate redundancies within the new XML/XHTML ecosystem, generalise some parts of XHTML into reusable XML extensions and specify the rest in a way that integrates nicely with everything else (leaving the decision what XML extensions to support up to the implementors).

The real game changer in semantics were Microformats[MFORM]. Because the W3C was too slow to keep up with the rapid evolution of the real web and too academic to be aware of real world problems, the web community came up with ways to use the existing building blocks of HTML to annotate markup in a machine-readable way, building on existing specs -- all this without XML.

When browser vendors finally gave up relying on the W3C's leadership for HTML and began working on a new HTML spec[HTML] under WHATWG, the main lesson was to pave the cowpaths. Instead of relying on authors to explicitly add annotations (although that is still possible using ARIA[ARIA]) various semantic elements like MAIN and ASIDE finally made it into the language. But the most important change was that the spec finally defined the error handling browsers had previously implemented inconsistently.

But again, even with the new HTML spec, HTML's goal never was to be able to declare semantics for any possible document. Yes, you can embed metadata (and the W3C still likes to push RDF-XML as a solution to do that in HTML) but the core elements are only intended to be good enough to provide sufficient semantics to structure generic documents.

Domain-level semantics are still left as an exercise to the reader. And anyone who's tried to actually parse well-formed, valid, well-structured HTML for metadata can tell you that even for generic documents the HTML semantics just aren't sufficient.

Sorry for the lengthy history lesson, but it never ceases to amaze me how rose-tinted some people's glasses are when looking at the mythical Semantic Web, early HTML and the XML ecosystem. I didn't even go into how much of a mess the latter is but I think my point stands even so.

[0]: The initial list of HTML tags[TAGS] allowed for "at least six" levels of heading but the RFC[HTML2] only defined h1-h6, but it also explicitly stated how each level was to be rendered. The original SGML guide[SGML] the tag list was loosely based on used up to six levels of headings in its examples. So in a nutshell, the reason there are six is likely a mix of "good enough" and nobody being able to figure out how to meaningfully define the presentation for anything beyond h6.

[SGML]: [http://cds.cern.ch/record/997909/files/cer-002659963.pdf](http://cds.cern.ch/record/997909/files/cer-002659963.pdf)

[TAGS]: [https://www.w3.org/History/19921103-hypertext/hypertext/WWW/...](https://www.w3.org/History/19921103-hypertext/hypertext/WWW/MarkUp/Tags.html)

[HTML2]: [https://tools.ietf.org/html/rfc1866](https://tools.ietf.org/html/rfc1866)

[HTML32]: [https://www.w3.org/TR/REC-html32](https://www.w3.org/TR/REC-html32)

[HTML4]: [https://www.w3.org/TR/REC-html40-971218/](https://www.w3.org/TR/REC- html40-971218/)

[XHTML1]: [https://www.w3.org/TR/2002/REC- xhtml1-20020801/](https://www.w3.org/TR/2002/REC-xhtml1-20020801/)

[XHTML11]: [https://www.w3.org/TR/2010/REC- xhtml11-20101123/](https://www.w3.org/TR/2010/REC-xhtml11-20101123/)

[XHTML2]: [https://www.w3.org/TR/2010/NOTE- xhtml2-20101216/](https://www.w3.org/TR/2010/NOTE-xhtml2-20101216/)

[MFORM]: [http://microformats.org/](http://microformats.org/)

[HTML]: [https://html.spec.whatwg.org/](https://html.spec.whatwg.org/)

[ARIA]: [https://www.w3.org/WAI/intro/aria](https://www.w3.org/WAI/intro/aria)

<comment><USER5> > as XML had much more well-defined implementation requirements, this among > other things meant that web pages that weren't well-formed XML would have to > break by definition and browsers were only allowed to show a yellow page of > death with the syntax error.

Indeed, part of the problem with switching to XHTML is that, even today, much HTML is authored by hand. On top of that web pages are often a mix of content from various sources. Third party widgets (like Twitter or ads) might be injected into the page. Or user content, which has often been converted by a hodgepodge of complex regexes into something vaguely resembling HTML.

</comment></comment><comment><USER6> I built a similar service called [https://Page.REST](https://Page.REST) couple of months back, which uses CSS selectors instead of the XPath for capturing content.

<comment><USER7> What about scraping based on logged in users?

</comment></comment><comment><USER8> I think it's what in french we call a <i>fausse bonne idée</i>, or good idea at first sight only.

I scrapped many site, and making an api out of a site is really similar to web scrapping. Despite having tried many time to create an universal scrapper, or at least a helper class for scrapping, I found out that restarting from scratch at each site was the best strategy.

There is a lot of info on stackoverflow about that, and there was a post specifically on this issue but i can't find it.

Maybe one third of the time, everything is ok and I use my favorite tools : python-requests with requests_cache for connection and lxml for parsing. Then, Toapi should work. But there is so much variety in internet site that most of the time, i end up doing something else : Use other way to get the data (websocket, phantomjs, chromeheadless, or going through proxy or Tor or anything ...). Use other data extraction methods, because xpath does not works (parsing json, making screenshot then ocr to get position of text, even fasttext !). Some time there is encoding issue. Some time html is malformated. In this 2/3 of the case, Toapi won't work out of the box. So you will trying to fix it, improve it, etc, and the code base of Toapi will grow but it will never handle all the case.

<comment><USER3> > (parsing json, making screenshot then ocr to get position of text, even > fasttext !)

Really interested to hear how you use fasttext for this?

<comment><USER8> I'll make a HN post about that. Though, don't hold your breath until it happens.

</comment></comment></comment><comment><USER9> This kind of service is great and technically well done. Of course it "should not be neccessary" if people exposed well-structured information in (X)HTML. However, many actors work hard to publish their valuable data only in a way they can control (for instance human-readable). Such API efforts are helpful to demonstrate that it is impossible to publish data and keep them fenced at the same time.

</comment><comment><USER10> I wish something like this would use asyncio rather than flask

</comment><comment><USER11> somehow misses an option to use local variables, i.e. use the id of the item to look up related data instead of only info inside of that element.

for the hacker news example that would be needed to parse score and comment count

</comment><comment><USER7> Does anyone know a service that does scraping of content behind login gates?

</comment><comment><USER12> This is still somewhat far from what I was hoping to see, but nonetheless a great inspiration and a good start.

I worked on an a small Firefox plugin whereby would enable visually impaired users to use firefox and interact with websites using voice. It was a small attempt and it was not an easy task...

The two biggest challenges were (1) understand the semantics of the website and (2) interacting with browser.

For challenge one, we can look at the case whereby a div/span/css clickable button is used instead of using the <button> tag. Traditional reader has tried its best to figure out how to find these buttons, but nonetheless it still posts a challenge. "alt" attribute is missing in <imh> tag so no description when image goes 404. So wouldn't it be awesome if we can be more responsible and also provide a standardized markup API so screen readers can use?

So here comes ARIA (Accessible Rich Internet Application) [1]. But even so a lot pf popular websites are not compliant woth ARIA. Furthermore, dynamic contents, single page application, and javascript makes screen reader more difficult to use.

As part of the thesis, I worked woth a few visually impaired users. Whether screen readers have evolved I do not know, so I's love to hear feedback from those on HN.

Since this project is intended to run in a browser, so the native choicenis an add-on (which mostly means Javascript). Furtheremore, to protect privacy, I want to do all the speech recognition locally, without leaving the user's local computer. So I tackled on Gmail and my college's website first.

Parsing Gmail's DOM was proven to be very difficult, but fortunately, Gmail.js [2] exists - I owe the author so much (although I did help fix a bug I think).

Next comes the interaction. The idea is

1\. user says to the computer "find the latest email"

2\. the computer repeats back to the user

3\. a small wait time is given to user a chance to correct the commmand before too late.

4.when time expires the code executes the command

Because there is no AI and because we neednto consider states, the best choice (esp for such small experiement), I created a finite state machine.

However, the number of states have exploded as expected for just 5-6 (??) commands for single website; clearly neither scalable nOr maintainable. It is unacceptable to limit a user to just a couple actions right?

What would be awesome is if every website can expose APIs - not just any API, but follows standard. To start small, we will take sitemap.xml as an inspiration. There should be a manifest file to describe how to login, what are needed to login, is there a "About us" page, how to do search, how to post something and etc.

This manifest file is generated on demand and is dynamic since we cannot write out 10,000 different interactions on Facebook righr? Then screen reader or my plugin can read the manifest file, and let user tell us what he/she desires to do, and we look up the manifest on how to make the call.

Toapi seems to be heading a similar direction, which is great.

The second challenge addon can only do so much. I did have to use the more low-level APIs to do certain thing, and if I remembered correctly service workers weren't exactly easy back in 2013. Content script has limited capability. Also, there was no speech recognition API in Firefox in 2013, so I turned to speak.js [3] which was the only one that worked offline and has no extra dependencies like nodeJS. There is some support for Web Speech API in Firefox which is encouraging [4].

The take aways are:

\- we suck at designing a usable website that respects semantics (I am guilty of that on my own silly homepage)

\- those don't use ARIA should try. ARIA does not solve everything but it gives a good start

\- DOM is a horrible place to use to deduce and find contrnts, and a nightmare to interact with

\- addon has restricted capability for good security reason so not a great idea to implement a complex software

\- javascript can hinder screen reader from doing its job - honestlt, please stop single page application and please stop hijacking the god damn back button

\- we hate ads espeically those overlay and popup ads (they cannot be fully blocked even with Chrome's new feature). But also consider we'd have to unblock ads to load some of the popular websites (ironically HN search will not work on mobile unless Focus is disabled)

\- finally, would be awesome if there is a way for machine to consime a website or a webpage's functionality

1\. [https://developer.mozilla.org/en- US/docs/Web/Accessibility/A...](https://developer.mozilla.org/en- US/docs/Web/Accessibility/ARIA)

2\. [https://github.com/KartikTalwar/gmail.js/tree/master](https://github.com/KartikTalwar/gmail.js/tree/master)

3\. [https://github.com/kripken/speak.js/](https://github.com/kripken/speak.js/)

4\. [https://hacks.mozilla.org/2016/01/firefox-and-the-web- speech...](https://hacks.mozilla.org/2016/01/firefox-and-the-web-speech-api/)

</comment><comment><USER13> Is this any different from any other framework such as Symfony that allows to create regex routes with a controller?

<comment><USER14> 1\. This is HTML DOM XPath-based, not regex based.

2\. It's a service, so you don't have to automate the repetition/automation part, and you get caching/discoverability/documentation/etc for free.

</comment></comment><comment><USER15> No support for SQL, to quickly query through complicated data structures?

<comment><USER16> I mean....that's a pretty arbitrary thing to assume would be present when the data structure is HTML. xpath, css selectors, etc... are the standards there.

<comment><USER15> So let's say I want to access a huge database; it's too large to fit in memory. Then why wouldn't SQL be an appropriate way to navigate through it?

<comment><USER17> You should use this to translate HTML into data, insert that data into a database, then do SQL.</comment></comment></comment></comment>


Deep Bilateral Learning for Real-Time Image Enhancement - <AUTHOR> https://groups.csail.mit.edu/graphics/hdrnet/ <comment><USER2> [http://halide-lang.org/](http://halide-lang.org/) is pretty good at optimising image filters for realtime use on mobile devices.

What neural networks are really good at, is if feature engineering the transform is difficult or time consuming. Like upscaling resolution (SRGAN) - or increasing dynamic range of LDR images by training with LDR-HDR pairs would be another nice use case. Neural nets for processing 1080p+ images have too many parameters to run well on mobile devices, but looks like this research gets around that (for some use cases).

Will have to play with the repo!

Film emulation (beyond the usual 3D LUTs for colour matching film stock) would be a fun use case. Wonder how much training data is required

<comment><USER3> Film emulation sounds like a special case of style transfer. Those run from a single image, so it might be reasonable to emulate it with very little data.

<comment><USER2> I think accurate film emulation would require a fair amount of training material pairs (digital/film) to learn the transformation between colours, colour/scene dependent dynamic range compression, and other artefacts like local contrast. The paper mentions using 4000 training pairs for their HDR+ example

</comment></comment></comment><comment><USER4> Buried lead is the awesome demo - [https://youtu.be/GAe0qKKQY_I?t=130](https://youtu.be/GAe0qKKQY_I?t=130)

<comment><USER5> I really would like to see them try different learning sets that vary the "styles" of retouching. This example looks like it's strongly biased to the _" make the images pop!"_ style of retouching, blowing highlights, shadows and contrasts.

What if the input set has more subtle retouching that pulls highlights and pushes shadows, but without the aforementioned issues?

What if they got their hands on the unedited and edited magnum photos? That would produce an interesting B&W filter, for sure!

[https://www.slrlounge.com/magnum-photos-darkroom-magic- genes...](https://www.slrlounge.com/magnum-photos-darkroom-magic-genesis- photoshop-lightroom/)

<comment><USER6> I wonder how many images are required to train a network like this?

If it's in the millions, getting pre and post retouching image pairs in such a quantity is likely impractical.

<comment><USER5> Ah, right, that could also be a limitation.

Although I'm pretty sure _Magnum Photos_ has a large quantity of images, but perhaps not all in a consistent style.

</comment></comment></comment></comment><comment><USER7> I think this paper is showing that you "can" train an auto exposure/white balancing/edit flow algorithm with a DL pipeline, but the results do not necessarily mean it will outperform simple and cheaper auto exposure/white balancing algorithms that's out there. And the flexibility in this approach also allows masking and background removal.

However, most of the examples in the paper in fact shows improvements of exposure and color. If you import those images and tweak 3 or 4 adjustments of clarity, curves, exposure, saturation in Polarr or Lightroom, you will quickly get very close to the result produced by this paper. However, it is still impressive that it could get to an exposure histogram that looks exact like the ground truth.

Maybe someone can benchmark this against the Google photos auto enhance. A lot of people turn the auto-enhance in Google off because it sometimes create unnatural looks for photos, which are tolerable to everyday consumer but for pros it just looks bad.

Lastly, if you look very closely on the input images, some of them appears to be artificially adjusted to show how the model works. (last page, 4th row, fist image, which looks both underexposured and overexposured after damping brightness through post processing), and these input images are not always the type of images you can get from cameras.

</comment><comment><USER8> Link to github repo is 404'ing ([https://github.com/google/hdrnet](https://github.com/google/hdrnet))

<comment><USER9> They have added a tool tip. Now, it is saying coming this week

</comment></comment><comment><USER10> "Enhancement" meaning tone-mapping? Are neural networks really required for that? Seems like a lot of heavy machinery for the resulting filter, but maybe tone-mapping standards have gone up.

<comment><USER3> They imply "human operator" level retouching, so potentially some combination of tone mapping, unsharp mask, edge enhancement, etc. as a single NN operation. It's also <30ms for 1080p on mobile, so potentially better than average speed.

<comment><USER6> This is effectively "tone mapping" which is aware of context. Ie. a face and a shoe might have the exact same color, yet they can end up different colours after processing even if they appear in the same image.

</comment></comment></comment><comment><USER11> this is the most complicated histo-stretch I've ever seen

</comment><comment><USER12> Very interesting approach. And it is always great to see teams provide actual pre-trained models. The less work people have to put in to reproduce your claims, the more likely you are to be taken seriously.

The code unfortunately returns a 404 for now. Hopefully, that is fixed soon.

</comment><comment><USER13> I find the examples of face brightening to detract from my impression of the entire work. Those images look so awkward, and are such poor photography that I'm not sure why someone would want to emulate them.</comment>


DestructHub: We want collaborators and people who loves challenges - <AUTHOR> https://github.com/DestructHub/ProjectEuler <comment><AUTHOR> The proposal of organization is a dump of each solution of common and bizarre challenges presents in programming world. Why? Because is fun, because is usefull for learning and we can learn with other people! Now we can be small, about one year old and we have about 20 members.

We want more polyglot programmers possible. Actually we have solutions in Python, C, C++, Go, Haskell, Elixir, Lua, Clojure and Common Lisp!

If you're interested, you can send a email to me: firstname.lastname@example.com

You love math? We love too! Actually we focusing on Project Euler since the creation! We have almost 100 solutions (counting the alternative solutions on all programming languages)!

Join to us and enters on the sea of destruction!</comment>