Harley-ml/lesswrong huggingface dataset:
# The Pleasures of Rationality

There are many pleasant benefits of improved [rationality](/lw/31/what_do_we_mean_by_rationality/):

*   [Winning](/lw/7i/rationality_is_systematized_winning/) more often.
*   Better [affective forecasting](http://en.wikipedia.org/wiki/Affective_forecasting).
*   Better [self-help skills](/lw/3nn/scientific_selfhelp_the_state_of_our_knowledge/) (e.g., [CBT](http://en.wikipedia.org/wiki/Cognitive_behavioral_therapy) is applied rationality).

I'd like to mention two other benefits of rationality that arise when working with _other_ rationalists, which I've noticed since moving to Berkeley to work with [Singularity Institute](http://intelligence.org/) (first as an intern, then as a staff member).

The first is the comfort of knowing that people you work with agree on literally _hundreds_ of norms and values relevant to decision-making: the laws of logic and probability theory, the recommendations of cognitive science for judgment and decision-making, the values of broad consequentialism and x-risk reduction, etc. When I walk into a decision-making meeting with Eliezer Yudkowsky or Anna Salamon or Louie Helm, I notice I'm more relaxed than when I walk into a meeting with most people. I know that we're operating on [Crocker's rules](http://wiki.lesswrong.com/wiki/Crocker's_rules), that we all want to make the decisions that will most reduce existential risk, and that we agree on how we should go about making such a decision.

The second pleasure, related to the first, is the extremely common result of reaching [Aumann agreement](/lw/881/the_pleasures_of_rationality/54bc) after initially disagreeing. Having worked closely with Anna on both the [rationality minicamp](http://intelligence.org/blog/2011/06/21/rationality-minicamp-a-success/) and a forthcoming article on intelligence explosion, we've had many opportunities to Aumann on things. We start by disagreeing on X. Then we reduce knowledge asymmetry about X. Then we share additional arguments for multiple potential conclusions about X. Then we both update from our initial impressions, also taking into account the other's updated opinion. In the end, we almost always agree on a final judgment or decision about X. And it's not that we agree to disagree and just move forward with one of our judgments. We actually both agree on what the most probably correct judgment _is_. I've had this experience literally _hundreds_ of times with Anna alone.

Being more rational is a pleasure. Being rational in the company of other rationalists is even better. Forget not the [good news of situationist psychology](/lw/52g/the_good_news_of_situationist_psychology/).
# Help needed: German translation of the Singularity FAQ

At [lukeprog's suggestion](/lw/7s7/useful_things_volunteers_can_do_right_now/), I translated the [Singularity FAQ](http://intelligence.org/singularityfaq) into German. Unfortunately, Tripitaka, who volunteered to proofread, doesn't seem to have read my messages. As the translation is far from perfect, I ask German-speaking users to help with finalizing the text. Also, I have no objections if you'd like to upvote this post (and the comments of anyone who participates) as a way of [incentivizing this kind of work](/lw/7td/how_to_incentivize_people_doing_useful_stuff_on/4wkq).  
  
The whole text should be proofread at least once, but you could help a great deal just by going over the highlighted sections. It helps if you are knowledgeable on the subject. The document is on Google docs, editable by anyone. I have made a backup and I think you can revert changes Wikipedia-style, so edit away!  
  
[Here it is](https://docs.google.com/document/d/1Qs4tJV0ikNg__IxoP5CIshE-q1w6O51A-dWdTybTfNY/edit?hl=en_US).  
  
[Side-by-side version](https://docs.google.com/document/d/1r28X90tOpj-CjHVadFMUCw8TzX4RHu9xY67fAkqWZRQ/edit?hl=en_US) (comment only, changes should be made in the other version).  
  
[To-do-list](https://docs.google.com/document/d/1h8N5cej77piamKaV1bx1kJzrQU_SUiTCg1LHyzqkkxg/edit?authkey=CMWztfsL&hl=en_US&authkey=CMWztfsL).

Edit: Fixed links.
# [LINK] Being proven wrong is like winning the lottery 

Phil Birnbaum at _Sabermetric Research_ [writes](http://sabermetricresearch.blogspot.com/2011/10/being-proven-wrong-is-like-winning.html) about how people have things backwards; it's great to find out that you're wrong:

> Let's suppose you open a restaurant, and you're very successful, and people like your food. You're very proud of being a great chef. Then, someone tells you, correctly, that one of your appetizers, one that you think is one of your best, is actually pretty awful. Your customers hate it.  
>   
> Your first reaction might be to get defensive. But, again, you should be thrilled! Now you can fix that dish. Your food, your restaurant, your profit, and your reputation will all be better than before. It's almost the best thing that can happen to you. Being wrong is like winning the lottery!

...

> I guess my overall point is that any online discussion, even between people who violently disagree with each other, should be a co-operative venture. One of you is wrong, and you're working together to find out who. And, we should keep in mind that most of the benefit goes to the person who was actually wrong in the first place.  
>   
> When someone you respect, or someone who seems to be expert and knowledgeable, starts disagreeing with you, it's like you've stumbled upon a fistful of lottery tickets. Argue your position, yes, but don't get defensive, and keep an open mind. Sure, it might be that other guy who's wrong. But if you're really, really lucky, it'll be you.
# 2011 Less Wrong Census / Survey

The final straw was noticing [a comment](/r/discussion/lw/88v/less_wrong_couchsurfing_network/54rg) referring to "the most recent survey I know of" and realizing [it was from May 2009](/lw/fk/survey_results/). I think it is _well_ past time for another survey, so here is one now.

**[Click here to take the survey](https://docs.google.com/spreadsheet/viewform?formkey=dHlYUVBYU0Q5MjNpMzJ5TWJESWtPb1E6MQ)**

I've tried to keep the structure of the last survey intact so it will be easy to compare results and see changes over time, but there were a few problems with the last survey that required changes, and a few questions from the last survey that just didn't apply as much anymore (how many people have strong feelings on _Three Worlds Collide_ these days?)

Please try to give serious answers that are easy to process by computer (see the introduction). And please let me know _as soon as possible_ if there are any security problems (people other than me who can access the data) or any absolutely awful questions.

I will probably run the survey for about a month unless new people stop responding well before that. Like the last survey, I'll try to calculate some results myself and release the raw data (minus the people who want to keep theirs private) for anyone else who wants to examine it.

Like the last survey, if you take it and post that you took it here, I will upvote you, and I hope other people will upvote you too.
# [link]s Recent developments in life-extension

Progeria is a very rare disease which causes children to undergo symptoms extremely similar to rapid aging, and generally dying before the age of 20. Recent results suggest that a fairly cheap drug may help reduce the aging systems, and it is possible it may have similar effects on normal humans. [Lay summary by BBC](http://www.bbc.co.uk/news/health-15536744) and [actual article](http://hmg.oxfordjournals.org/content/20/20/3997.short)

On a related note, in a strain of mice which age unusually quickly, there's been success delaying many symptoms of aging by removing senescent cells. Here is the relevant [article in Nature.](http://www.nature.com/nature/journal/vaop/ncurrent/full/nature10600.html)
# AI Challenge: Ants

Aichallenge.org has started their third AI contest this year: [Ants](http://aichallenge.org/index.php).

> The AI Challenge is all about creating artificial intelligence, whether you are a beginning programmer or an expert. ... \[Y\]ou will create a computer program (in any language) that controls a colony of ants which fight against other colonies for domination. ... The current phase of the contest will end December 18th at 11:59pm EST. At that time submissions will be closed. Shortly thereafter the final tournament will be started. ... Upon completion the contest winner will be announced and all results will be publically available.
> 
> Ants is a multi-player strategy game set on a plot of dirt with water for obstacles and food that randomly drops. Each player has one or more hills where ants will spawn. The objective is for players to seek and destroy the most enemy ant hills while defending their own hills. Players must also gather food to spawn more ants, however, if all of a player's hills are destroyed they can't spawn any more ants.

I mentioned this in the [open thread](/lw/8ad/open_thread_november_2011/55js), and there was a discussion about possibly making one or more "official" LessWrong teams. D_Alex has offered a [motivational prize](/lw/8ad/open_thread_november_2011/55nc). If this interests you, please discuss in the comments!
# Selection Effects in estimates of Global Catastrophic Risk

Here's a poser that occurred to us over the summer, and one that we couldn't really come up with any satisfactory solution to. The people who work at the Singularity Institute have a high estimate of the probability that an Unfriendly AI will destroy the world. People who work for [http://nuclearrisk.org/](http://nuclearrisk.org/) have a very high estimate of the probability that a nuclear war will destroy the world (by their estimates, if you are American and under 40, then nuclear war is the single most likely way in which you might die next year). 

It seems like there are good reasons to take these numbers seriously, because Eliezer is probably the world expert on AI risk, and Hellman is probably the world expert on nuclear risk. However, there's a problem - Eliezer is an expert on AI risk _because_ he believes that AI risk is a bigger risk than nuclear war. Similarly, Hellman chose to study nuclear risks and not AI risk I _because_ he had a higher than average estimate of the threat of nuclear war. 

It seems like it might be a good idea to know what the probability of each of these risks is. Is there a sensible way for these people to correct for the fact that the people studying these risks are those that have high estimate of them in the first place?
# Tell me what you think of me

Time and time again, honest feedback has improved my life. I have sought it out on many specific occasions, but now I have a static, anonymous way for people to give me feedback — for any reason, at any time.

You can give me feedback on my personality, my conduct, or the organization for which I work [**by following this link right here**](http://tinyurl.com/luke-feedback).

Thank you. I apologize for making a discussion post that is all about me.

I operate by [Crocker's Rules](http://tinyurl.com/Crockers-rules).
# Q&A with new Executive Director of Singularity Institute

Today I was appointed the new Executive Director of Singularity Institute.

Because I care about transparency, one of my first projects as an intern was to begin work on the organization's first [Strategic Plan](http://intelligence.org/blog/2011/08/26/singularity-institute-strategic-plan-2011/). I researched how to write a strategic plan, tracked down the strategic plans of similar organizations, and met with each staff member, progressively iterating the document until it was something everyone could get behind.

I quickly learned why there isn't more of this kind of thing: transparency is a lot of work! 100+ hours of work later, plus dozens of hours from others, and the strategic plan was finally finished and ratified by the board. It doesn't accomplish much by itself, but it's one important stepping stone in building an organization that is more productive, more trusted, and more likely to [help solve the world's biggest problems](http://intelligence.org/overview/whyworktowardthesingularity).

I spent two months as a researcher, and was then appointed Executive Director.

In further pursuit of transparency, I'd like to answer (on video) submitted questions from the Less Wrong community [just as Eliezer did two years ago](/lw/1lq/less_wrong_qa_with_eliezer_yudkowsky_video_answers/).

**The Rules**

1) One question per comment (to allow voting to carry more information about people's preferences).

2) Try to be as clear and concise as possible. If your question can't be condensed into one paragraph, you should probably ask in a separate post. Make sure you have an actual question somewhere in there (you can bold it to make it easier to scan).

3) I will generally answer the top-voted questions, but will skip some of them. I will tend to select **questions about Singularity Institute as an organization**, not about the technical details of some bit of research. You can read some of the details of the Friendly AI research program in my [interview with Michael Anissimov](http://intelligence.org/blog/2011/09/15/interview-with-new-singularity-institute-research-fellow-luke-muehlhuaser-september-2011/).

4) If you reference certain things that are online in your question, provide a link.

5) This thread will be open to questions and votes for 7 days, at which time I will decide which questions to begin recording video responses for.

I might respond to certain questions within the comments thread and not on video; for example, when there is a one-word answer.
# Singularity Institute mentioned on Franco-German TV

The following is a clipping of a [documentary about transhumanism](http://www.arte.tv/de/suche/4135792.html) that I recorded when it aired on [Arte](http://en.wikipedia.org/wiki/Arte), September 22 2011.

At the beginning and end of the video Luke Muehlhauser and Michael Anissimov give a short commentary.

**Download here: [German](http://kruel.co/arte-siai.flv), [French](http://kruel.co/arte-siai-french.flv)** (ask for HD download link). Should play with [VLC player](http://www.videolan.org/vlc/).

  

Sadly, the people who produced the show seemed to be somewhat confused about the agenda of the Singularity Institute. At one point they seem to be saying that the SIAI believes into "the good in the machines", adding _"how naive!"_, while the next sentence talks about how the SIAI tries to figure out how to make machines respect humans.

Here is the original part of the clip that I am talking about:

> In San Francisco glaubt eine Vereinigung ehrenamtlicher junger Wissenschaftler dennoch an das Gute im Roboter. Wie naiv! Hier im Singularity Institute, dass Kontakte zu den großen Unis wie Oxford hat, zerbricht man sich den Kopf darüber, wie man zukünftigen Formen künstlicher Intelligenz beibringt, den Menschen zu respektieren.  
>   
> Die Forscher kombinieren Daten aus Informatik und psychologischen Studien. [Ihr Ziel](http://www.youtube.com/watch?v=0A9pGhwQbS0): Eine Not-to-do-Liste, die jedes Unternehmen bekommt, das an künstlicher Intelligenz arbeitet.

My translation:

> In San Francisco however, a society of young voluntary scientists believes in the good in robots. How naive! Here at the Singularity Institute, which has a connection to big universities like Oxford, they think about how to teach future artificial intelligences to respect humans.

I am a native German speaker by the way, maybe someone else who speaks German can make more sense of it (and is willing to translate the whole clip).
# Query the LessWrong Hivemind

Often, there are questions you want to know the answers to. You want other people's opinions, because knowing the answer isn't worth the time you'd have to spend to find it, or you're unsure whether your answer is right.

LW seems like a good place to ask these questions because the people here are pretty rational. So, in this thread: **You post a top-level comment with some question. Other people reply to your comment with their answers**. You upvote answers that you agree with and questions whose answers you'd like to know.

A few (mostly obvious) guidelines:

For questions:

*   Your question should probably be in one of the following forms:
    *   Asking for the probability some proposition is true.
    *   Asking for a confidence interval.
*   Be specific. Don't ask when the singularity will happen unless you define 'singularity' to reasonable precision.
*   If you have several questions, post each separately, unless they're strongly related.

For answers:

*   Give what the question asks for, be it a probability or a confidence interval or something else. Try to give numbers.
*   **Give some indication of how good your map is**, i.e why is your answer that? If you want, give links.
*   If you think you know the answer to your own question, you can post it.
*   If you want to, give more information. For instance, if someone asks whether it's a good idea to brush their teeth, you can include info about flossing.
*   If you've researched something well but don't feel like typing up a long justification of your opinions, that's fine. Rather give your opinion without detailed arguments than give nothing at all. You can always flesh your answer out later, or never.

This thread is primarily for getting the hivemind's opinions on things, not for debating probabilities of propositions. Debating is also okay, though, especially since it will help question-posters to make up their minds.

Don't be too squeamish about breaking the question-answer format.

This is a followup to [my comment](/r/discussion/lw/8ad/open_thread_november_2011/55oz) in the open thread.
# Low legibility of Cognitive Reflection Test dramatically improves performance?

I'm reading Kahneman's Thinking, Fast and Slow and I've stopped on this:

> 90% of the students who saw the CRT in normal font made at least one mistake in the test, but the proportion dropped to 35% when the font was barely legible. You read this correctly: performance was better with the bad font.

This seems like an important finding, but I can't find references in the book (Kindle) or on the Web. Does anybody know any real evidence for this claim? _EDIT: I found [the original paper](http://pages.stern.nyu.edu/~aalter/intuitive.pdf)_

Do you think that people could behave rationally with such a simple intervention?

[simple intro to CRT](http://whilecharliesleeps.blogspot.com/2006/03/take-cognitive-reflection-test.html)

_EDIT: fixed spelling in title_
# Why would an AI try to figure out its goals?

> "So how can it ensure that future self-modiﬁcations will accomplish its current objectives? For one thing, it has to make those objectives clear to itself. If its objectives are only implicit in the structure of a complex circuit or program, then future modiﬁcations are unlikely to preserve them. Systems will therefore be motivated to reﬂect on their goals and to make them explicit." -- Stephen M. Omohundro, [The Basic AI Drives](http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf)

> This AI becomes able to improve itself in a haphazard way, makes various changes that are net improvements but may introduce value drift, and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever). -- Eliezer Yudkowsky, [What I Think, If Not Why](/lw/wp/what_i_think_if_not_why/)

I have stopped understanding why these quotes are correct. Help!

More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?
# Which fields of learning have clarified your thinking? How and why?

Did computer programming make you a clearer, more precise thinker? How about mathematics? If so, what kind? Set theory? Probability theory?

Microeconomics? Poker? English? Civil Engineering? Underwater Basket Weaving? (For adding... _depth._)

Anything I missed?

Context: I have a palette of courses to dab onto my university schedule, and I don't know which ones to chose. This much is for certain: I want to come out of university as a problem solving _beast_. If there are fields of inquiry whose methods easily transfer to other fields, it is those fields that I want to learn in, at least initially.

Rip apart, Less Wrong!
# Sometimes, talking the issue through *works*

Michael Nielsen's new book _[Reinventing Discovery](http://www.amazon.com/Reinventing-Discovery-New-Networked-Science/dp/0691148902/)_ is invigorating. Here's one passage on how a small group talked an issue through and had a large impact on scientific progress:

> Why is it that biologists share genetic data in GenBank in the first place? When you think about it, it’s a peculiar choice: if you’re a professional biologist it’s to your advantage to keep data secret as long as possible. Why share your data online before you get a chance to publish a paper or take out a patent on your work? In the scientific world it’s papers and, in some fields, patents that are rewarded by jobs and promotions. Publicly releasing data typically does nothing for your career, and might even damage it, by helping your scientific competitors.
> 
> In part for these reasons, GenBank took off slowly after it was launched in 1982. While many biologists were happy to access others’ data in GenBank, they had little interest in contributing their own data. But that has changed over time. Part of the reason for the change was a historic conference held in Bermuda in 1996, and attended by many of the world’s leading biologists, including several of the leaders of the government-sponsored Human Genome Project. Also present was Craig Venter, who would later lead a private effort to sequence the human genome. Although many attendees weren’t willing to unilaterally make the first move to share all their genetic data in advance of publication, everyone could see that science as a whole would benefit enormously if open sharing of data became common practice. So they sat and talked the issue over for days, eventually coming to a joint agreement—now known as the Bermuda Agreement—that all human genetic data should be immediately shared online. The agreement wasn’t just empty rhetoric. The biologists in the room had enough clout that they convinced several major scientific grant agencies to make immediate data sharing a mandatory requirement of working on the human genome. Scientists who refused to share data would get no grant money to do research. This changed the game, and immediate sharing of human genetic data became the norm. The Bermuda agreement eventually made its way to the highest levels of government: on March 14, 2000, US President Bill Clinton and UK Prime Minister Tony Blair issued a joint statement praising the principles described in the Bermuda Agreement, and urging scientists in every country to adopt similar principles. It’s because of the Bermuda Agreement and similar subsequent agreements that the human genome and the HapMap are publicly available.
# List of potential cognitive enhancement methods

Participants in the Singularity Summit 2011 workshops held on October 17-18 brainstormed a list of cognitive enhancement methods they would like to see tested — some of them for the first time, many of them more thoroughly than has been done so far. Here is that list:

*   rationality instruction
*   potassium and nutrients/micronutrients in general
*   modafinil and its class
*   racetam class, adderall, riatlin
*   paleo and other popular diets
*   multigenerational embryo selection
*   particular TMS interventions
*   doing math or logic problems every day
*   amount of sleep 
*   neurofeedback; EEG, etc.
*   physical health, specific types of exercise, yoga
*   more red blood cells
*   certain types of electrical stimulation
*   spaced repetition
*   practicing visualization
*   practicing chess and Go
*   dual n-back training
*   prolonged sensory deprivation
*   experience manipulating physical objects (craftsmanship and engineering)
*   listening to music actively (and engaging with other art forms actively)
*   using specific visual languages for specific tasks
*   happiness in general (gratitude, etc.)
*   changing the oxygen content of what you're breathing
*   gene expression levels
*   operant conditioning
*   irradiation or administration of other toxins
*   lucid dreaming
*   GHB
*   intermittent fasting
*   creatine
*   nicotine, caffeine
*   arterial glucose drip to get more sugar to the brain
*   Ekman training
*   steroids
# New Q&A by Nick Bostrom

Underground Q&A session with Nick Bostrom ([http://www.nickbostrom.com](http://www.nickbostrom.com/ "http://www.nickbostrom.com")) on existential risks and artificial intelligence with the Oxford Transhumanists (recorded 10 October 2011).

[http://www.youtube.com/watch?v=KQeijCRJSog](http://www.youtube.com/watch?v=KQeijCRJSog)
# Babyeater's dilemma

Imagine it's the future, and everything has gone according to plan. Humanity has worked out its own utility function, _f~0~_, and has worked out a strategy _S~0~_ to optimize it.  
  
Humanity has also run a large number of simulations of how alien worlds evolve. It has determined that of those civilizations which reach the same level of advancement - that know their own utility function and have a strategy for optimizing it - there is an equal probability that they will end up with each of 10 possible utility functions. Call these _f~0~...f~9~_.  
  
(Of course, these simulations are coarse-grained enough to satisfy the [nonperson predicate](/lw/x4/nonperson_predicates/)).  
  
Humanity has also worked out the optimal strategy _S~0~...S~9~_ for each utility function. But they just happen to score poorly on all of the others:  
  
_f~i~(S~i~) = 10_  
_f~i~(S~j~) = 1_ for i != j  
  
In addition, there is a compromise strategy C:  
  
_f~i~(C) = 3_ for all i.  
  
The utility functions, _f~0~_ through _f~9~_, satisfy certain properties:

**They are altruistic**, in the sense that they care just as much about far-away aliens that they can't even see as they do about members of their own species.

**They are additive**: if one planet implements _S~j~_ and another implements _S~k~_, then:  
_f~i~(S~j~_ on one planet and _S~k~_ on the other_) = f~i~(S~j~) \+ f~i~(S~k~)_.

(This is just to make things easier - the problem I'm describing will still apply in cases where this rule doesn't hold).

**They are non-negotiable**. They won't "change" if that civilization encounters aliens with a different utility function. So if two of these civilisations were to meet, we would expect it to be like the [humans and the babyeaters](/lw/y4/three_worlds_collide_08/): the stronger would attempt to conquer the weaker and impose their own values.

In addition, humanity has worked out that it's very likely that a lot of alien worlds exist, i.e. aliens are really really real. They are just too far away to see or exist in other [Everett branches](http://wiki.lesswrong.com/wiki/Everett_branch).  
  
So given these not entirely ridiculous assumptions, it seems that we have a multiplayer [prisoner's dilemma](http://wiki.lesswrong.com/wiki/Prisoner%27s_dilemma) even though none of the players has any causal influence on any other. If the universe contains 10 worlds, and each chooses its own best strategy, then each expects to score 19. If they all choose the compromise strategy then each expects to score 30.  
  
Anyone else worried by this result, or have I made a mistake?
# Rationality Dojo Examples?

Early on in my exposure to Less Wrong I encountered the metaphor of [Rationality as Martial Art](/lw/gn/the_martial_art_of_rationality/). I assumed at some point I would be a member of an active Rationality Dojo, regularly training and becoming progressively more formidable as I learned the Art.

Several years later, though I meet regularly with an [awesome local group](https://www.facebook.com/pages/Less-Wrong-Ottawa/221226327934037) whose company I greatly enjoy, I still feel as though my training has not yet begun.

Can anyone point to an example of an active Rationality Dojo? What do you do there (Games? Exercises? [Kata](http://en.wikipedia.org/wiki/Kata)?)? Who are the [instructors](/lw/9c/mandatory_secret_identities/)? The closest examples that I've seen are the [Mega](/lw/4wm/rationality_boot_camp/)\- and [Mini](/lw/5ec/minicamp_on_rationality_awesomeness_and/)-camps; can anyone shed some additional light on what went on there?
# Open Research 

This is a [TED talk](http://www.ted.com/talks/michael_nielsen_open_science_now.html) about open science. It starts with a description of a new math problem which is offered on a blog, and which eventually attracts enough mathematicians working on it to solve, not just the original problem, but a more difficult version of it. It was enough easier than the usual way of doing math that it was described as being like driving a car instead of pushing it.

Then the speaker talks about more ambitious projects-- like a wiki about quantum computing-- which get started, but no one is actually willing to do the work, so that the wiki lies all but vacant.

He suggests that public science isn't what scientists get paid for nor what builds their careers, and has some ideas for pushing the standards of science to change. There's been at least [one success](http://en.wikipedia.org/wiki/Bermuda_Principles) involving publishing genomes.

Perhaps the reason the math project succeeded was because the problem was small enough that success was both well-defined and possible, not to mention that working on it was probably more fun than figuring out how to do tolerable and sensibly-linked wiki articles.

There _may_ be a way to get publicly funded science to be open source. We're already got proof of concept for solving math problems if they're interesting enough, so I suggest going public if you've got a math problem people might like to work on.
# Stanovich, 'The Robot's Rebellion' (mini-review)

The jacket text for Keith Stanovich's _[The Robot's Rebellion](http://www.amazon.com/Robots-Rebellion-Finding-Meaning-Darwin/dp/0226771253/)_ sums up the book well:

> The idea that we might be robots is no longer the stuff of science fiction; decades of research in evolutionary biology and cognitive science have led many esteemed scientists to the conclusion that... humans are merely the hosts for two replicators (genes and memes) that have no interest in us except as conduits for replication...
> 
> Accepting \[this\] disturbing idea, Keith Stanovich here provides the tools for the "robot's rebellion," a program of cognitive reform necessary to advance human interests over the limited interest of the replicators and define our own autonomous goals as individual human beings. Drawing on the latest research... _The Robot's Rebellion_ describes how short-term and reflexive thinking processes dominate the higher-order thinking necessary for achieving autonomy from our biological programming. These higher-order evaluative activities of the brain... hold the potential to fulfill our need to ascribe significance to human life.
> 
> We may well be robots, but we are the only robots who have discovered that fact. \[This\] is the first step in constructing a radical new concept of self based on what is truly singular about humans: that they gain control of their lives in a way unique among life forms on Earth — through rational self-determination.

The book is an excellent introduction to the first stage of Yudkowskian philosophy: We are robots in a mechanistic universe running on a swiss army knife of cognitive modules. But at least we finally _noticed_ we're robots, and we can use the skills of rationality to hop off our habit treadmills and pursue our values instead. These values are complex and often arbitrary, but we can use our reflective capacities to extrapolate our values based on "higher-order" desires, a desire for preference consistency, and other considerations. All this is argued for at length in Stanovich's book. The only thing missing is a discussion of what to do about all this when AI arrives.
# New AI risks research institute at Oxford University

The [Oxford Martin Programme on the Impacts of Future Technology](http://www.futuretech.ox.ac.uk/) (aka FutureTech) is a new research department at Oxford University, roughly a spin-off of FHI, but _focusing_ on AI and nanotech risks and [differential technological development](http://en.wikipedia.org/wiki/Differential_technological_development). Like FHI, this department is directed by [Nick Bostrom](http://nickbostrom.com/). They'll be hiring more researchers soon. Basically, this means more people and money being devoted to existential risk reduction.

![](http://commonsenseatheism.com/wp-content/uploads/2011/11/party.gif)

Okay, now back to work.
# Is latent Toxoplasmosis worth doing something about?

_[Toxoplasmodi gondii](http://en.wikipedia.org/wiki/Toxoplasma_gondii)_ is a parasitic protozoa who's primary host is cats but also infects other mammals, primarily mice and rats but including humans, as part of its life cycle. Infection by _Toxoplasmodi gondii_ is called [Toxoplasmosis](http://en.wikipedia.org/wiki/Toxoplasmosis) and may be acute (flu like symptoms) or latent. 

Toxoplasmosis is extremely common. Worldwide, about [30% (US 11%; France 88%!)](http://en.wikipedia.org/wiki/Toxoplasmosis#In_humans) of people about of people have Toxoplasmosis.

Toxoplasmosis is known to cause [behavioral changes in rats](http://en.wikipedia.org/wiki/Toxoplasmosis#Biological_modifications_of_the_host):

> It has been found that the parasite has the ability to change the behaviour of its host: infected rats and mice are less fearful of cats—in fact, some of the infected rats seek out cat-urine-marked areas. This effect is advantageous to the parasite, which is able proliferate if a cat eats the infected rat and thereby becomes a carrier.

Observational studies suggest that latent Toxoplasmosis may also cause behavioral changes in humans ([source paper](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2526142/pdf/sbl074.pdf)). The observed differences between infected people and non-infected people include:

> *   Decreased novelty seeking behavior
> *   Slower reactions
> *   Lower rule-consciousness and greater jealousy (in men)
> *   Greater warmth, conscientiousness and moralistic behavior (in women)

It's also suspected by some of being [a cause of Schizophrenia](http://en.wikipedia.org/wiki/Toxoplasmosis#Toxoplasma.27s_role_in_schizophrenia).

Obviously some or all these may be due to unobserved 3rd causes.

There haven't been any randomized studies yet, as far as I know. It seems like such studies would be easy to conduct rigorously since a high fraction of the population is already infected. For example, by finding people who are already infected and randomly cure some of them. This kind of experiment is even pretty close to how you would expect the information to be used. 

I've been around cats a fair amount and the base rate is high in the US, so my chances of having latent Toxoplasmosis seem fairly high. Thus I am curious whether this is worth doing something about. Diagnosis sounds like it is fairly simple (PCR on blood samples). It's easy enough so that it can be done in large scale studies at least. [Treatment](http://en.wikipedia.org/wiki/Toxoplasmosis#Latent) is done with atovaquone and clindamycin, which appear to be relatively inexpensive.

I'd expect the effects to be net negatives (most random changes are detrimental) and even if the behavioral effects are smallish, the effects over a lifetime will add up. Has anyone else gotten tested and/or treated for latent Toxoplasmosis? Is it worth it?
# Three more classes coming from Stanford of interest here

[http://www.pgm-class.org/](http://www.pgm-class.org/) \- Probabilistic Graphical Models

[http://www.nlp-class.org/](http://www.nlp-class.org/) \- Natural Language Processing

[http://www.game-theory-class.org/](http://www.game-theory-class.org/) \- Self explainatory  
  
ETA:

[http://infotheory-class.org/](http://infotheory-class.org/)
# OPERA Confirms: Neutrinos Travel Faster Than Light

> [New high-precision tests carried out by the OPERA collaboration in Italy broadly confirm its claim, made in September,](http://news.sciencemag.org/scienceinsider/2011/11/faster-than-light-neutrinos-opera.html) to have detected neutrinos travelling at faster than the speed of light. The collaboration today submitted its results to a journal, but some members continue to insist that further checks are needed before the result can be considered sound.

**Link:** [nextbigfuture.com/2011/11/faster-than-light-neutrinos-opera.html](http://nextbigfuture.com/2011/11/faster-than-light-neutrinos-opera.html)

> The OPERA Collaboration sent to the Cornell Arxiv an updated version of their preprint today, where they summarize the results of their analysis, expanded with additional statistical tests, and including the check performed with 20 additional neutrino interactions they collected in the last few weeks. These few extra timing measurements crucially allow the ruling out of some potential unaccounted sources of systematic uncertainty, notably ones connected to the knowledge of the proton spill time distribution.
> 
> \[...\]
> 
> **So what does OPERA find ?** Their main result, based on the 15,233 neutrino interactions collected in three years of data taking, is unchanged from the September result. The most interesting part of the new publication is instead that the  find that the 20 new neutrino events (where neutrino speeds are individually measured, as opposed to the combined measurement done with the three-year data published in September) **confirm the earlier result: the arrival times appear to occur about 60 nanoseconds before they are expected.**

**Link:** [science20.com/quantum\_diaries\_survivor/opera\_confirms\_neutrinos\_travel\_faster_light-84763](http://www.science20.com/quantum_diaries_survivor/opera_confirms_neutrinos_travel_faster_light-84763)

**Paper:** [kruel.co/paper-neutrino-velocity-JHEP.pdf](http://kruel.co/paper-neutrino-velocity-JHEP.pdf)

**Previously on LW:** [lesswrong.com/lw/7rc/particles\_break\_lightspeed_limit/](/lw/7rc/particles_break_lightspeed_limit/)
# Where do selfish values come from?

Human values seem to be at least partly selfish. While it would probably be a bad idea to build AIs that are selfish, ideas from AI design can perhaps shed some light on the nature of selfishness, which we need to understand if we are to understand human values. (How does selfishness work in a decision theoretic sense? Do humans actually have selfish values?) Current theory suggest 3 possible ways to design a selfish agent:

1.  have a [perception-determined utility function](/lw/116/the_domain_of_your_utility_function/) (like AIXI)
2.  have a static (unchanging) world-determined utility function (like UDT) with a sufficiently detailed description of the agent embedded in the specification of its utility function at the time of the agent's creation
3.  have a world-determined utility function that changes ("learns") as the agent makes observations (for concreteness, let's assume a variant of UDT where you start out caring about everyone, and each time you make an observation, your utility function changes to no longer care about anyone who hasn't made that same observation)

Note that 1 and 3 are not reflectively consistent (they both refuse to pay the [Counterfactual Mugger](http://wiki.lesswrong.com/wiki/Counterfactual_mugging)), and 2 is not applicable to humans (since we are not born with detailed descriptions of ourselves embedded in our brains). Still, it seems plausible that humans do have selfish values, either because we are type 1 or type 3 agents, or because we were type 1 or type 3 agents at some time in the past, but have since self-modified into type 2 agents.

But things aren't quite that simple. According to our current theories, an AI would judge its decision theory using that decision theory itself, and self-modify if it was found wanting under its own judgement. But humans do not actually work that way. Instead, we judge ourselves using [something mysterious](/lw/2id/metaphilosophical_mysteries/) called "normativity" or "philosophy". For example, a type 3 AI would just decide that its current values can be maximized by changing into a type 2 agent with a static copy of those values, but a human could perhaps think that changing values in response to observations is a mistake, and they ought to fix that mistake by rewinding their values back to before they were changed. Note that if you rewind your values all the way back to before you made the first observation, you're no longer selfish.

So, should we freeze our selfish values, or rewind our values, or maybe even keep our "irrational" decision theory (which could perhaps be justified by saying that we intrinsically value having a decision theory that isn't too alien)? I don't know what conclusions to draw from this line of thought, except that on close inspection, selfishness may offer just as many difficult philosophical [problems](/lw/1mo/the_preference_utilitarians_time_inconsistency/) as altruism.
# Where do I most obviously still need to say "oops"?

Eliezer once told me:

> The most common error I see on Less Wrong is the failure to say "[Oops](/lw/i9/the_importance_of_saying_oops/)."

If there's one rationality skill I like to think I'm pretty good at, it's this one: the skill of saying "Oops."

In fact, I say "Oops, fixed, thanks" so often on Less Wrong I once [suggested](/lw/63i/rational_romantic_relationships_part_1/55aj) I should have a shortcut for it: "OFT."

And I don't just say "oops" for typos and [mistakes](/lw/78s/help_fund_lukeprog_at_siai/4q7f) in [tone](/lw/74f/are_deontological_moral_judgments_rationalizations/4nle), but also for mistakes in my [facts](/lw/71x/a_crash_course_in_the_neuroscience_of_human/4ogq) and [arguments](http://commonsenseatheism.com/?p=1174).

It's not that I say "oops" every time I'm _challenged at length_, either. I don't say "oops" until I _actually_ think I was significantly wrong; otherwise, I [stand](/lw/899/great_explanations/55pc) my [ground](/lw/4vr/less_wrong_rationality_and_mainstream_philosophy/3qe4) and ask for better counter-arguments.

But I'm sure I can improve.

Wanna help me debug my own mind?

Tell me: On which issues do you think I most obviously still need to say "Oops"?
# How to prove anything with a review article

Thus the subtitle of [this blog posting](http://blogs.plos.org/obesitypanacea/2011/11/21/cigarettes-may-be-useful-for-distance-runners-or-how-to-prove-anything-with-a-review-article) at PLoS, referencing [this article](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3001541/) on "Cigarette smoking: an underused tool in high-performance endurance training". The point being that you can write a review article to argue anything you want, with sufficient cherry-picking and chains of links.

If you are doing actual experiments and making observations or proving theorems, then to a large extent -- larger in some sciences than in others -- you are constrained by the brute facts. But when writing secondary literature, especially in areas where data is generally fuzzier, it is easy, whether deliberately or not, to write to a bottom line, including findings you like and excluding those you don't.

Something to bear in mind when reading or writing any review article.
# 'Next post' and 'Previous post' links for posts in a sequence

Less Wrong would be [stickier](http://www.phoenixrealm.com/how-to-increase-your-website-stickiness/) if there were links at the bottom of each post in [The Sequences](http://wiki.lesswrong.com/wiki/Sequences) to the next and previous posts in that sequence.

I just added those links for each post in my own sequences: [The Science of Winning at Life](http://wiki.lesswrong.com/wiki/The_Science_of_Winning_at_Life), [Rationality and Philosophy](http://wiki.lesswrong.com/wiki/Rationality_and_Philosophy), and [No-Nonsense Metaethics](http://wiki.lesswrong.com/wiki/No-Nonsense_Metaethics).

I can't do that for sequences written by somebody else. Perhaps one or more of the [LW editors](/lw/5kt/who_are_the_lw_editors/) would be willing to start hacking away on that project?

Here's the algorithm I executed:

1.  Open all the posts from one sequence, in order, in browser tabs.
2.  Go to first post in the sequence.
3.  Click 'Edit'.
4.  Click 'HTML' and uncheck 'Word Wrap.'
5.  Scroll to the bottom of the post (not counting notes and references) and paste in the following:  
    
    <p>&nbsp;</p>  
    <p align="right">Next post: <a href=""></a></p>  
    <p align="right">Previous post: <a href=""></a></p>  
    <p>&nbsp;</p>  
    <p>&nbsp;</p>
    
6.  If post is first in sequence, remove 'Previous post' line.
7.  If post is last in sequence, remove 'Next post' line.
8.  Paste in URL and post title for remaining 'Next post' or 'Previous post' lines of HTML.
9.  Click 'Update', then click 'Submit'.
10.  If this is the last post in the sequence, return 0. Else, move to next post in sequence and go to step #3.
# Video: Skepticon talks

The [talks](http://skepticon.org/schedule.php) from [Skepticon](/lw/8fn/skepticon_iv_meetup_planning/) [IV](/lw/8ho/meetup_skepticon_iv_meetup_saturday_night/) are being [posted to YouTube](http://www.youtube.com/user/HamboneProductions#p/u). 

So far we have:

*   [Richard Carrier on Bayes](http://www.youtube.com/watch?v=HHIz-gR4xHo) (my favorite)
*   [Julia Galef on the Straw Vulcan](http://www.youtube.com/watch?v=tLgNZ9aTEwc) 
*   [Greta Christina on angry atheists](http://www.youtube.com/watch?v=GUI_ML1qkQE&feature=related)
*   [Hermant Mehta on math education](http://www.youtube.com/watch?v=bx2Agfhjv20&feature=related)
*   [David Fitzgerald on Mormonism](http://www.youtube.com/watch?v=DTi8dq4KAeE&feature=related)
*   [J.T. Eberhard on mental illness](http://www.youtube.com/watch?v=UI-YvrHZVvk&feature=related) (a dramatic end to the conference)
*   [an "atheist revival" by Sam Singleton](http://www.youtube.com/watch?v=bRpzwVAgOpI&feature=related) (on the lighter side)

ADDED:

*   ["Death Panel"](http://www.youtube.com/watch?v=Cd0yF2ucFbE) featuring Julia Galef, Eliezer Yudkowsky, Greta Christina, and James Croft
*   [Darrel Ray on secularism and sex](http://www.youtube.com/watch?v=83P6epN1e9w&feature=related)
*   [Eliezer Yudkowsky on heuristics and biases](http://www.youtube.com/watch?v=TwqYB1uzcU4) (really more like a crash course in the core LW sequences)
*   [Joe Nickell on paranormal investigations](http://www.youtube.com/watch?v=RAAjcUrhido) (I missed this at the conference; and even more regrettably, missed the chance to ask Joe Nickell [what he thinks of many-worlds](/lw/1kh/the_correct_contrarian_cluster/1crn).)
*   [Jen McCreight on "skeptical genetics](http://www.youtube.com/watch?v=mf5ZcpgZop4)" (the other talk I missed)
*   [Rebecca Watson on the religious right](http://www.youtube.com/watch?v=f6VXNTZYsyk)
*   [Spencer Greenberg on self-skepticism](http://www.youtube.com/watch?v=wW_oNxax5RQ)
*   [Dan Barker on atheist clergy](http://www.youtube.com/watch?v=dup6xkvj1S0)

More to come soon, hopefully...