jkeisling/hacker-news-corpus-2007-2022 huggingface dataset:
There's a lot of stuff under the "Big Data" umbrella: I focused on Hadoop below because that's my focus right now. I'm sure I'm missing some roles here, but the specialities I can think of are:

- getting data out of production systems and transforming it (infrastructure or ETL)
- analytical querying and reporting
- system administration
- machine learning

There's also the wide world of NoSQL data stores, which people lump in with big data, but which require vastly different skills.

The Hadoop VM I linked to above is good for working through exercises for all of the above.

As a starting point, this book[1] walks through the motivation behind Hadoop, and then gets a little into internals and use cases. It's out of date, but you can work through it and get into the right frame of mind, understand HDFS, etc. It's a good starting point.

AMP Camp (that I linked to above) is an introduction to Spark for people with a little Hadoop experience. Spark is getting a lot of attention, you could run into it in a number of roles.

If you're going to be planning the whole pipeline, or doing any sort of infrastructure role, I recommend Hadoop Application Architecture[2] for more modern tools and design patterns. This blog post[3] is a pretty good overview of distributed logs, which are essential for horizontal scale. Understanding Kafka and ZooKeeper is really useful for infrastructure roles, maybe less so for admins.

If you're planning to be in the reporting layer, having a deep understanding of SQL and data warehousing is useful. This book[4] is old hat, but I would say it's expected knowledge for anyone planning a warehouse, and it's interesting to understand best practices. Most places will also expect knowledge of Tableau or a similar BI tool, but that's tougher to learn on your own since licenses are brutal. Visualization with D3 is nice to have in this space, especially if you're coming from a web background - Scott Murray's tutorials [5] are a good starting place.

It's harder to point to resources for sysadmins - if you weren't a sysadmin before, you need to understand a lot of other concepts before you worry about Hadoop stuff. ML is similar - you need to understand the principles and be able to work on a single node. There's lots of good resources out there about getting started in data science.

1. http://shop.oreilly.com/product/0636920021773.do

2. http://shop.oreilly.com/product/0636920033196.do

3. http://engineering.linkedin.com/distributed-systems/log-what...

4. http://ca.wiley.com/WileyCDA/WileyTitle/productCd-0471200247...

5. http://alignedleft.com/tutorials
Core.async is an official Clojure and ClojureScript library that takes channels from the Go language and adds them to Clojure in a much nicer way.

http://clojure.com/blog/2013/06/28/clojure-core-async-channe...

All of the operations are expressions (not statements)

This is a library, not syntax

alts! is a function (and supports a runtime-variable number of operations)

Priority is supported in alt

--

Quil is a Clojure and ClojureScript library for creating interactive drawings and animations using Processing (https://en.wikipedia.org/wiki/Processing_%28programming_lang...).

--

Om is actually an interface to Facebook's React. Ironically, the terminology, and the feature set of Om has little, if any, resemblance to React. Om specifically transforms Clojure data so that it can be represented in a way that React can uderstand.

In the Clojure world, we build on top of React and stay away from anything that resembles JavaScript and it's behavior. React provides a virtual DOM and we just use Clojure to talk to the rendering engine.

https://gist.github.com/runexec/6efb322f29fc790a98a1

--

Chestnut allows live coding in the browser. This means all changes are visible without having to reload the page.

2 Minute video of Om using Chestnut. https://www.youtube.com/watch?v=gI3fJKmvgq4

https://github.com/om-cookbook/om-cookbook

--

Apache Cordova is the Apache License version of Adobe PhoneGap. ClojureScript compiles to highly optimized JavaScript already, so you can make mobile apps using ClojureScript and Apache Cordova.
Repetition does wonders to the perceived truth value of a story.
> I probably don't even have the tools installed (unlike with ruby)

Provided correct versions are in place.

In my aforementioned observations, Gem installation was complicated when the parent app relied on gem versions below the code currently in GitHub.

This is of course a problem not unique to Ruby (see: Default Python version on CentOS), based on the number of versioning utilities for Go. I have also had plenty of problems with autoconf's that don't work.
I don't understand. What malice exactly is being asserted in the parent comment?
"[It] was not till 1956 that [Nepal] set [their] watches for the first time to Nepal Standard Time, with the meridian at Mt Gauri Shankar, 100km east of Kathmandu [capital of Nepal]. It wasn't Mt Everest because Gauri Shankar was closer to Nepal's centre of gravity, as it were.

It was a choice that set [Nepal's] clocks 10 minutes ahead of India, which at the time used the longitude that passed through Calcutta. When [India] switched their meridian to Hyderabad in 1971, [Nepal] officially had four degrees of separation, and presto, found ourselves a further five minutes ahead of the Indians."

[1]http://web.archive.org/web/20110725100005/http://www.nepalit...
>Are we beginning to see the tipping point where mainstream corporates are starting to question whether traditional OO is the right way to go?

Nope. http://www.tiobe.com/index.php/content/paperinfo/tpci/index....

First functional language on that list is F# at 19.
No one knows about xsel anymore?
I love community ideas like this and the ability for anyone off the street to rapidly prototype a product/widget...so cool!
It makes me wonder whether it is a sign that desktop responsiveness has been neglected by the kernel devs which possibly prioritize server issues. I had read a Google engineer suggesting Canonical should hire decent kernel developers : " P.S. Next thing for Ubuntu to learn --- how to pay their engineers
well enough, and how to give them enough time to work on upstream
issues, that once they gain that experience on Ubuntu's dime and
become well known in the open source community, they don't end jumping
ship to companies like Red Hat or Google. :-)"

http://news.ycombinator.com/item?id=1321029
That office is pretty minimalist!

It only looks busy because of the clutter which is not part of the office design so much as just some stuff he hasn't found time to clean up.

The clutter is all temporary stuff for work, not permanent features of the office.
IETester works for me.
Those same skills could be taught by teaching a course in discrete math and probability. The latter would give people more useful life skills.
Thanks for the link to Rich Hickey's post. How software development should be funded is something I've been thinking about for a while now, and I hadn't come across that link yet. He gives a good summary of the existing situation. (For anyone interested, it's also reprinted on the Clojure website [1].)

I tend to agree with the points that Bradley Kuhn makes in one of his blog posts [2], but I'm still trying to figure out how direct payment for time spent would work for non-obvious scenarios, like end-user desktop or mobile apps.

[1]: http://clojure.org/funding
[2]: http://www.ebb.org/bkuhn/blog/2010/07/07/producing-nothing.h...
Why the downvotes? This morning after they launched this, you could once again add apps and they'd actually show up in iTunes Connect.
I'm fascinated that in the UK, software engineers are not viewed as white-collar professionals. That seems to be a rather big difference between the work culture of the UK and the USA.
I was curious about how it behaved with locked profiles. The authentication dialog that popped up was not very attractive, and I certainly didn't want to enter my password in it.
> If your immediate superior harasses you for doing the job you are being paid for you need to go over their head.

His problem isn't with his superiors, it's with his co-workers:

> If you don't stare at boring stuff on the monitor for 8 hours straight, co-workers suspect you being a slacker. Yes, not the managers who see the output. Only the co-workers who see the process and can't relate to this kind of work.
There is another possible explanation, though it sounds even more like an airport novel than the Stuxnet story does.

Consider that a lot of people, including the US, Israel, and just about everyone else on the planet would like to do anything possible to derail the Iranian nuclear activities. Making the centerfuges self-destruct by introducing Stuxnet would do that - but making the Iranians think that a virus was the problem would have the same effect. I'm sure that the Iranians spent a lot of effort trying to remove the malware from the computers controlling the centerfuges and they probably even took them off-line until they were sure that the controllers were clean. Why would you risk the very valuable remaining centerfuges until you were sure the problem was fixed? They might have even tried to replace the Windows controllers with something running Linux or Solaris or something. It's quite likely that the clean-up from Stuxnet delayed the enrichment process even more than the original damage.

Malware is a convenient excuse for computer problems. "I don't have my homework because the computer crashed" certainly sounds valid but it's also an excuse that people immediately understand and have a hard time disputing. The Iranians may have internally blamed Stuxnet for the problems but isn't it possible that someone at Natanz might be using that to off-load blame for some other problem?

It's also a convenient story for Israeli and Western Intelligence. Perhaps the Israelis really did do something to destroy the centerfuges - perhaps they had a mole on the inside who unbalanced the centerfuges with sweaty fingerprints or grains of sand. It's reasonably likely, especially this close to the time of the event, that the story they would want the Iranians to believe isn't the actual story. Put out a false account and watch the Iranians running around reformatting their computers while your man on the inside stays safe and ready to act again.

If the Stuxnet story is true it shows remarkable technical cleverness by whoever did it. But if they're that clever they might be even slightly more cunning - enough to throw off the Iranians as to the actual cause and retain the ability to bung up the works again.
When you are first starting out white-label offers can be enticing to say the least. I have not heard of any companies doing them recently though. Most of the time, white label is common in the hosting/web-based service (i.e. email/project management/support companies).

Like ecommando said, don't get sucked into it though, unless you can control every aspect of the deal, code, database, etc... As the white-label grows they will expect the world from you in terms of support/updates/code modifications/etc... They will take away from your time to support your own customers. I would offer a hybrid solution at most where you allow them to change the logo and remove references to your company, but you continue to host and control the code.

To answer your question on why they wouldn't pay/use, well it is all about marketing. They want to be able to offer something different from their competitors and make it look like they are the ones behind it.

As for pricing, NEVER, NEVER take a lump sum payment. Always work something from a revenue-share or monthly fee arrangement. You could even do an annual contract with built-in price increase each year based on percentages of their sales/revenue.

If your product is that good that other companies are approaching you to white-label the solution, the best advice I can give you is to review your marketing/sales strategy. Why can't you sell direct to their market? Why risk losing revenue and allowing them to make money off your product?
Wired has always been the worst kind of tabloid (The Web is Dead, push technology is the future, the dotcom bubble won't ever burst...), their lack of journalistic ethics doesn't surprise me in the least.
I see what you mean. It definitely needs to be fixed on Firefox. Thanks a lot.
I used to have to think a great deal before typing, with Haskell and ML. Nowadays, I should think a great deal, but the thinking goes on at a higher level in the design than I'm used to. Of course, I could be prototyping.

But the point is, there is light at the end of the tunnel. No one told me about that. These were languages that were supposed to discourage wild hacking. And they do. But that doesn't keep you from prototyping, or alternately, writing bad APIs.
That, or our brains have hit some biological equivalent of Moore's law ending for silicon computing. Maybe getting a doubling in brain size would require a quadrupling in brain energy consumption (and power dissipation) at our level.
This. I've realized years ago that the current fiat system can be unsustainable because it's very hard for regulators to avoid the temptation of applying QE and other monstrosities. But I realized way too late that BTC is the vehicle to flee from that (in a simple way compared to moving physical assets across borders).

It already succeeded in helping rich people in China to bring their wealth out of control. This has maximum utility for a lot of people. I'm not sure what will come after the current financial system (and it most probably won't go anywhere and will co-exist for some years before crypto will be banned everywhere), but this new type of wealth transfer mechanism already gets used a lot. Everyone who distrusts the current financial system or is oppressed by their regime is interested in it.

I don't know how the world will look like (it'll probably be very hard to continue the current path because capital has found a new way to move across borders and avoid taxes, again) and I don't know if it's generally good for global society, but Pandora's box is open.

I agree with the energy discussion and that it's wasteful, but the utility of it (transacting without governmental permission) seems to be higher for the participants. And they don't care about its socioeconomic impact, they care about its utility for them (money for miners and speculators who keep BTC's value high and the network functional, permissionless transaction for the actual target audience / users of BTC). BTC was built for permissionless money transactions, the speculation is just a byproduct and everyone is focusing on it instead of looking at its actual value for its users.

edit for the sibling comments: You need BTC only to move your assets from one country/wallet to another. You can then move it to a stable coin like USDC and slowly and safely get it out of the crypto system without having to deal with the volatility. This takes you half an hour and you're done. It's currently an investment vehicle, but its utility comes from being able to get money from A to B regardless of government intervention. Even if BTC drops to $3k, it still has this value. Looking at the price of BTC is missing the point: it's more than $0 and you can do a transaction in 10 minutes and that is all that matters to move assets.
This only works if you slice and dice the polity. Who says zoning in the Bay Area should be decided by current residents? The super commuter service workers from Modesto should certainly have a say. So should the families displaced to Phoenix or Las Vegas.

In addition, it is not a fair characterization to describe apartment buildings as "a shot in the nuts."
My personal favorite that I've seen is FLAMINGASS (Facebook Lyft Amazon Microsoft Intel Netflix Google Apple Salesforce Square) (at least one of these is definitely a bit of a stretch)
Don't worry. I bet there are lots of HN'ers who regularly experiment and A/B test HN comments & submissions, in order to learn what yields the best karma. A bit like SEO but just for HN.
To illustrate, here is a video that doesn't work on your iPad or iPhone.

I don't mean that to come across as snarky. I'm just a little sad that I couldn't watch it sitting here on my back porch.
Wesabe has an API, but I believe the documentation is offline. See https://www.wesabe.com/groups/50-wesabe-api-developers/discu... for a bit more on that.
I'm really surprised no one else is discussing this point. The article says that potassium stays mostly in solution in the body, which means that regardless of how many bananas you ate (or anything else with high levels of potassium) your body is just going to excrete most of it.

So there is no real comparison to make between bananas and any dose of gamma+beta radiation, making the whole "banana equivalent dose" at best mistaken and at worst willfully misleading.
Get used to it. You aren't a founder - or at least haven't been presented as one (hmmm...) To the rest of the world you are no better than the guy who empties the trash. This is also what your boss Ben thinks but he's probably smart enough to keep his opinion to himself - as long as you're making him rich.
If (like me) you didn't know what to make of this, the video of the talk by Raymond Hettinger is here:
http://www.viddler.com/explore/python-italia/videos/85/
He starts with this example at ~9:30 in, and boy is it cool...
I have been to some European countries, and that's patently untrue. Residential neighborhoods tend to contain lots of shared housing -- apartments/rowhouses with shared walls between units -- and all attendant noise and unpleasantness.

The only residential neighborhood I ever enjoyed was when I stayed with a family who lived as I do: four vehicles for four people, in a freestanding home, driving to the grocery daily. The major difference in our lifestyles is that, ignoring exchange rates, they paid easily five times as much to achieve my standard of living. I don't think they would willingly trade places with their middle class employees, living in the crowded row housing.

To put it another way, I reject your underlying argument. Yes, hundreds millions of Europeans have become accustomed to a lower standard of living than mine. No, that does not mean I will happily acclimate myself to the same standard.
For some context, Git follows the same architecture as Codeville, so Linus didn't invent the idea (although he ripped it off from Monotone, not Codeville) and the argument was essentially about whether a simple three-way merge can be used in all cases, and the answer is no, because of criss-cross cases, and solutions for that have since been put into Git. It is the case that semantics which more closely resemble three way merge are preferred though, as explained in my post which this thread is about, but for reasons which nobody in the flame war you link to appreciated at the time.
Thanks for the kind feedback. It's always very much appreciated :-)
Old news. And there are similar plugins:

histwin: https://github.com/chrisbra/histwin.vim

undobranche_viewer.vim: http://www.vim.org/scripts/script.php?script_id=2141
It's a cheaper way to do it but yeah it does have issues with weather and all. (cables getting cut by trees etc..) The major reason for not doing it underground is probably due to building rights and such which might be more trouble than they are worth for google at this point. Plus the conduits are expensive, might be cheaper to just put new cables when some break than building conduits.
Right, but this is my point, unless there is a compelling reason for the technology of Windows to run Node.js over nix, most competent Windows dev teams will choose a hybrid environment and use nix to run Node.

So perhaps I am being short sighted and there is a compelling technical reason to run Node.js on Windows, but if it's just having a market of devs who don't want to develop on *nix then I don't see it.
Will the radiation pose challenges to the electronic equipment on the robots?
Changes to a site require a significant amount of data before it is really meaningful. Looking at the response and actions of an individual and then trying to apply that to how the masses will respond almost always ends in failure.

While looking at a specific visitor's actions on your website might be interesting, it's not useful in the grand scheme of things. Interacting with them via Olark is different, since you have the opportunity to directly influence their decisions.

Have you ever been watching your real time analytics and been able to react to them in real time, to the benefit of you (the website owner) or the website visitor? I'm not trying to be antagonistic, just trying to determine the real benefit of real-time analytics and how they are being used.
It’s a very cute classroom project – but to really be accepted means the government will have it’s hands on the flow, revenue and taxes from said bitcoin just as with the dollar bill. So besides this being anything more than a social experiment – its about as smart as buying POGS if people remember those things...
I'm studying Spanish, so I made a little script that fetches Calvin & hobbes cartoons in both Spanish and English and then converts each strip into bilingual strip with ImageMagick. It works well enough (>99%).

#!/bin/bash
for i in {1..1000}; do x=$(date +'%Y/%m/%d' -d "$i days ago");
z=$(date +'%Y-%m-%d' -d "$i days ago");
wget https://www.gocomics.com/calvinandhobbesespanol/$x -O $z-es;
wget https://www.gocomics.com/calvinandhobbes/$x -O $z-en;
y=$(cat $z-es | grep -o -P '.{0}https://assets.amuniversal.com.{0,33}' | tail -1);
f=$(cat $z-en | grep -o -P '.{0}https://assets.amuniversal.com.{0,33}' | tail -1);
wget $y -O $z-es.gif; rm $z-en; sleep 1;
wget $f -O $z-en.gif; rm $z-es; sleep 1;
convert $z-en.gif $z-es.gif -append $z-bi.gif; #ImageMagick
rm $z-es.gif;
rm $z-en.gif;
done;
OTOH the mRNA vaxs can mutate as fast as the virus.
Why is this astrometeorlogical paper in a journal about human physiology?
Direct audio: https://www.podtrac.com/pts/redirect.mp3/audio.wnyc.org/radi...
Thanks for sharing!

With the creation and proliferation of Modern Hebrew, there was a cost, though. Yiddish, Judeo-Arabic (the language Maimonides wrote a lot of literature in), and Ladino were victims of the centralization of Hebrew as the common language for Jews.

Yiddish is a German dialect, Judeo-Arabic an Arabic dialect, and Ladino is a Spanish dialect and are all dying/dead languages with attempted revival movements in their own right, but nothing with too much steam and this article lays out a potential reason why; aside from traditional, historic, or cultural reasons, there's no real societal reason for these languages in today's world.
Thanks for sharing! :)

Yes, the feature computation on the images has to be sone only once and the representation is very efficient - 512 float16 values per image.
Dont forget this nice article: https://bisqwit.iki.fi/story/howto/dither/jy/
Its such a statement about the free rider problem that python can have only 2 people working fulltime on it. Python is one of the most popular languages in the world - there's got to be trillions of dollars worth of companies depending on it. Small improvements to the language would result in millions of dollars of productivity gains in aggregate globally.

And yet ... collectively we can only find enough money to pay two full time salaries?! Seriously?
Please no. I liked C. Watching it die the death by a thousand cuts that is C++ featuritis is painful.